Filing against -bad because I am realistic :)

Warning:

This proposal assumes correcly behaving elements. It isn't concerned with
closed / open GOP as it deals with already decoded frames.

Problem:
--------

One wants to be able to query "recently" decoded frames from a pipeline without
needing to decode them again.

One wants to be able to scrub forward *and* backward in an "as smooth as
possible" way.

Use cases:
----------

For now, in such a pipeline:

decodebin ! some_filter random_property=X ! sink

if one wants to update random_property in the PAUSED state and
get the same frame updated in the sink, one has to seek flush+accurate.

That's hardly ideal.

In the paused state again, if one wants to step to the next frame,
same deal, flush+accurate, decodebin crunches the same thing over again.

Proposed solution:
------------------

On modern-day workstations, a lot of RAM is readily available.
A full HD decoded picture in the RGBA colorspace is 8294400 bytes, ie roughly
7.91 megabytes.
On my machine with 16 GB of RAM, I'd happily sacrifice 1GB, keeping in mind that unused RAM is bad RAM.

This represents around 130 frames that could be stored.

The proposal here is to implement a framecache element to act as storage, active only in the PAUSED state, in passthrough mode in the PLAYING state.

This element would be inserted immediately after the decodebin.

One could imagine using multiple frame caches after each filter to maximize smoothness, but this would introduce complexity to decide who would handle the seek, as far as I can tell changing properties of elements in the decodebin can not affect the actual output image (or can it ?), whereas changing a property on a filter can.

Seek handling "negotiation" is thus out of scope and excluded of this proposal.

Part of the complexity in the proposed design will be due to the requirement of 
having both forward *and* backward scrub seeking working smoothly.

How to ensure that the first seek to a given region remains as fast as without the framecache, and at the same time store frames as soon as possible in the surrounding region.

Solution: the frame cache intercepts the seek, and returns TRUE. Caller is responsible for checking the bus for ASYNC_DONE, or if the seek actually failed, a custom "async seek failed" that need not be defined in this proposal. (does this already exist ?)

All the following operations take place in a task running on the sinkpad of the framecache.

The framecache element will then request a
SEEK_KEY_UNIT | GST_SEEK_FLAG_SNAP_BEFORE upstream, which should be as fast
as a normal seek (check -> is it true ?). start and stop are set at the required start.

Upon reception of the segment, the element knows where the previous keyframe was, takes note of that. It also knows the next position before which to seek when filling itself in the backward direction.

It then waits for EOS and pushes the last buffer it received downstream. All the buffers are stored, a counter is incremented to mark the number of buffers stored before the current position.

The element then requests a SEEK_KEY_UNIT | GST_SEEK_FLAG_SNAP_AFTER (note that some time is "wasted" in that first operation, ideally one would be able to specify "KEY_UNIT | SNAP_BEFORE" for the start and "KEY_UNIT | SNAP_AFTER" for the stop in the first seek but meh.

Buffers are stored blabla.

stop is noted, one now knows the next position to seek when filling the buffer in the forward direction. A counter is incremented to mark the number of buffers stored after the current position.

Things go on until the sum of "before" and "after" buffers exceeds a "soft limit".

When one receives a new seek, two solutions are possible:

1) The buffer is already stored in the framecache's memory -> it is sent, counters for "before" and "after" buffers are updated,
if needed some buffers are discarded on one side and seeks performed on the other side to maintain "symetry".

2) The buffer is not stored in the framecache's memory: immediately forward a new flushing seek upstream, repeating the process that happened on the first seek, the same update as in the other case is performed.

Flushes of course have to be handled appropriately by the element, this isn't detailed here.

As it is an element with only one sink and one source pad possibilities for race conditions seem low at first sight ^^