Here is the pipeline I use for testing:

GST_DEBUG=videoaggregator:7 gst-launch-1.0 -ev compositor wait-sinks=none name=comp sink_1::alpha=0.5 sink_1::xpos=50 sink_1::ypos=50 ! fakesink silent=false  videotestsrc pattern=snow timestamp-offset=3000000000 ! "video/x-raw,format=AYUV,width=640,height=480,framerate=(fraction)30/1" ! queue2 ! comp.   videotestsrc pattern=smpte timestamp-offset=2000000000 ! "video/x-raw,format=AYUV,width=800,height=600,framerate=(fraction)10/1" ! queue2 ! comp.

I think a enumeration property would be better. I think there is valid use case where you want an intermediate mode, like waiting for at least 1 frame, while you proposal is to wait for all sink pads to have a frame.

p.s. you could indent your examples, it's unreadable

I think this should be a per-sinkpad property in GstAggregator, it's not only useful for video.

(In reply to Sebastian Dröge (slomo) from comment #3)
> I think this should be a per-sinkpad property in GstAggregator, it's not
> only useful for video.

That clash with my idea that it would be nice if the aggregator could start when the first chunk of data arrive, instead of producing background meanwhile.

This issue might be related (or not) to this one : https://bugzilla.gnome.org/show_bug.cgi?id=749517

(In reply to Philippe Renon from comment #5)
> This issue might be related (or not) to this one :
> https://bugzilla.gnome.org/show_bug.cgi?id=749517

Not entirely. This is about having the ability to choose the first frame behaviour.

- Produce first frame on deadline (background/silence if nothing came in)
- Produce first frame only when at least one pad has data
- Produce first frame only when all pad have data

And because there is 3 decent uses cases for each, I think this is worth an enumeration to select the first-frame mode.

(In reply to Nicolas Dufresne (stormer) from comment #6)
> (In reply to Philippe Renon from comment #5)
> > This issue might be related (or not) to this one :
> > https://bugzilla.gnome.org/show_bug.cgi?id=749517
> 
> Not entirely. This is about having the ability to choose the first frame
> behaviour.
> 
> - Produce first frame on deadline (background/silence if nothing came in)

In live pipelines we always must produce frames on deadline, no matter what. But the property could influence if we actually wait until the deadline, or if we already produce data before if some pads have data.

> - Produce first frame only when at least one pad has data
> - Produce first frame only when all pad have data

This is only about the very first frame, and after that you wait as usual? That doesn't sound right.

How does 3) is differ from 1), considering that for live streams we always must produce data on deadline, and that for non-live streams the deadline is infinity (i.e. we produce data when all pads have data).



I still think a per-pad configuration makes more sense for this. Usually from the application side you know which pads will have data when, which are important to wait for and which not.


Also I think it would make sense in aggregator to calculate the running time of the segment.start of each pad, and don't wait for the pads where the running time of segment.start is smaller than the current output running time.
(Maybe not aggregator, but audio/videoaggregator only).

(In reply to Sebastian Dröge (slomo) from comment #7)

> I still think a per-pad configuration makes more sense for this. Usually
> from the application side you know which pads will have data when, which are
> important to wait for and which not.

Also for the "global settings" (produce when one pad has data), you're completely dependent on your OS's scheduler what the frames contain. And it might have interesting effects in video/audioaggregator, if due to scheduler jitter we produce gaps. E.g.

> stream 1:
>   [0, 1] [0, 2] [0, 3]
> stream 2:
> [0, 1]     [0, 2]   [0, 3]

Here you would with the "produce when one pad has data" setting produce lots of gaps in the stream.

I think we are confusing a first frame strategy from run-time strategy. The original use case here is that following:

v4l2src name=a ! mixer.
v4l2src name=b ! mixer.
  videomixer name=mixer background=black ! output

Currently, the first few frame will randomly be:

  black | a | a+b | ...
  black | b | a+b | ...
  a | a+b | ...
  b | a+b | ...
or
  a+b | a+b

Because v4l2src may take different amount of time to start producing after the pipeline has reached playing state. At run-time, it won't make much difference, the current mixer behaviour is fine and the matching is quite smooth.

The problem is that this deadline only behaviour result in random mix of streams all the time. The use case, and I think it's a fair live use case, is that we would prefer having a initial gap then having a first frame which is the background color defined in the mixer (or only 1 of the two images).

What I'm proposing here, is to enable for multiple first frame strategy (instead of 2 using a boolean). So we can extend as needed. I think this discussion already raised 4 cases.

1. Only start produce if a+b is possible (data on all pad, could wait forever if the source can be sparse, but that is not a problem in live editing)
2. Only start producing if at least one of a or b is possible
3. Only start produce if a is present (or any other, that one would need something on the pad indeed)
4. Produce as soon as the next deadline is met. This is current behaviour.

Also, you can't put this in GstAggregator directly, as it's really specific to raw formats, it wouldn't make sense when we use GstAggregator as a base class for muxers.

I noticed another interesting problem here. Aggregator always outputs running time 0 to infinity. Now if you add e.g. compositor later to the pipeline (or to a live pipeline for that matter), you will have lots of too late frames with nothing in the beginning.

I have this problematic use case:

compositor name=mixer sink_1::ypos=50 ! videoconvert ! timeoverlay shaded-background=true auto-resize=false ! autovideosink

v4l2src ! mixer.

udpsrc port=9000 ! identity dump=false ! textrender halignment=left line-alignment=left ! video/x-raw, width=320, height=120 ! videorate drop-only=true ! video/x-raw, framerate=10/1 ! mixer.

The pipeline will not start until a 1st udp packet is sent to the udpsrc.
Once that packet is sent, the pipeline will complain about lots of too late frames. After that it runs fine.

This issue was first declared in https://bugzilla.gnome.org/show_bug.cgi?id=749517 (sorry to mention it again...).
Again I am not entirely sure "my" issue is related to the issues discussed here.
Don't hesitate to tell me to go away if it is not :)

@Philippe In git master, it will start immediately, regardless if there is a frame from v4l2src or udpsrc. What is being discussed here is the ability to bring back the old behaviour, which may be wanted in certain cases.

@Nicolas Yes you are right the compositor and the pipeline start immediately.
I am not up to par with the people here... and the gstreamer vocabulary.

Anyways, the issue I am facing is that the compositor starts but its chain method will block until all the sink pads have received at least one buffer.

I'd like the compositor to start producing composited frames as soon as one pad has a buffer. It is useful when compositing live sources and non live sources.
Currently, live sources are "blocked" until all non live ones have produced one buffer (can be very long in my case).

Does this issue belong here or is it covered by https://bugzilla.gnome.org/show_bug.cgi?id=749517 ?

@Sebastian You have reworded the other issue description to
 "aggregator: need support for sparse streams".
I am not sure I fully understand what sparse streams are. My definition of sparse would be "mostly empty" or "with holes". While a sparse stream that has an initial empty segment (?) will stall the compositor, any source that takes time to produce a 1st buffer will also stall it.
Could also affect live sources in a way. When using two or more webcams, the compositor might stall until the slowest one has produced a frame (not tested...).

To conclude : I feel that the issue I am describing is somewhat related to the topic under discussion here but might be wrong...
For now, I’ll stop derailing this conversation ;)

(In reply to Philippe Renon from comment #14)

> I'd like the compositor to start producing composited frames as soon as one
> pad has a buffer. It is useful when compositing live sources and non live
> sources.
> Currently, live sources are "blocked" until all non live ones have produced
> one buffer (can be very long in my case).

That's not correct (or would be a bug). If there is at least one live source going into aggregator, it will only be blocked until the time for the media has come. If any streams don't provide buffers early enough for that, they will be ignored.

There's just the problem I described in comment 11 here.

> Does this issue belong here or is it covered by
> https://bugzilla.gnome.org/show_bug.cgi?id=749517 ?
>
> @Sebastian You have reworded the other issue description to
>  "aggregator: need support for sparse streams".

That was a misunderstanding of your problem :) Sparse streams are exactly what you describe, for example subtitle streams. I think this here (not the other bug) is very related to your problem.

Maybe Philippe case is that his input stream has no framerate, in which case it will indeed block. This is not fully linked to the intent of this bug and shall be discussed in another bug please.

(In reply to Nicolas Dufresne (stormer) from comment #16)
> Maybe Philippe case is that his input stream has no framerate, in which case
> it will indeed block. This is not fully linked to the intent of this bug and
> shall be discussed in another bug please.

Ok. I'll keep an eye on the issue tracked here and if it does not address mine then I'll reactivate that other issue.

PS : but I believe that my pipeline has a frame rate on the two streams.
Here it is again:

compositor name=mixer sink_1::ypos=50 ! videoconvert ! timeoverlay shaded-background=true auto-resize=false ! autovideosink

v4l2src ! mixer.

udpsrc port=9000 ! identity dump=false ! textrender halignment=left line-alignment=left ! video/x-raw, width=320, height=120 ! videorate drop-only=true ! video/x-raw, framerate=10/1 ! mixer.

Created attachment 305318 [details] [review]
aggregator: Add property to select how to decide on a start time

Before aggregator based elements always started at running time 0,
now it's possible to select the first input buffer running time or
explicitly set a start-time value.

Created attachment 305319 [details]
app.c

sample application that adds a compositor to a live pipeline after 2 seconds.

Play around a bit with the properties, and also switch to audio :)

Frédéric, Philippe, does this also solve your problem?

Created attachment 305327 [details] [review]
aggregator: Add property to select how to decide on a start time

Before aggregator based elements always started at running time 0,
now it's possible to select the first input buffer running time or
explicitly set a start-time value.

(In reply to Sebastian Dröge (slomo) from comment #21)
> Frédéric, Philippe, does this also solve your problem?

No, my use case is to encode and store the composite of 2 live sources.
So what I need is to wait for all sinks and start outputting at the last timestamp. For example, using compositor, I have 2 streams starting respectively at 300ms and 400ms. The output stream should start at 400ms to avoid having black frames or incomplete composition.

Maybe it could correspond to a 4th option GST_AGGREGATOR_START_TIME_SELECTION_LAST ?

Yeah, that would be some kind of "last". However that's non-trivial to implement :) You would have to drop input buffers (in live mode only!) until all streams had at least one buffer, and from that point start outputting. In non-live mode you can block until all pads have buffers, and then just take the highest start timestamp of all pads.

Feel free to add that on top of my patch

(In reply to Sebastian Dröge (slomo) from comment #22)
> Created attachment 305327 [details] [review] [review]
> aggregator: Add property to select how to decide on a start time

This patch does not work well for live elements when using the "first" setting. Before the first buffer is received, elements will report that the "next time" is 0. Then 0 becomes the deadline and we start outputting things from 0.

Has to be extended to really not timeout until the first buffer is actually received. Doing that later.

Created attachment 305709 [details] [review]
aggregator: Add property to select how to decide on a start time

Before aggregator based elements always started at running time 0,
now it's possible to select the first input buffer running time or
explicitly set a start-time value.

I haven't tested the patch yet but I don't think that it will address "my" issue.
My issue is that a pipeline will stall in compositor until all sink pads have received a 1st buffer (which can be never in some cases).

Yeah, your case is something orthogonal to this then. Yours is about selecting the pads for which aggregator should wait (in non-live mode).

(In reply to Sebastian Dröge (slomo) from comment #26)
> Created attachment 305709 [details] [review] [review]
> aggregator: Add property to select how to decide on a start time

This one also does not completely do the right thing yet. It would wait for all pads to have data for the first aggregation, instead of just waiting for the first pad.

Created attachment 308343 [details] [review]
aggregator: Add property to select how to decide on a start time

Before aggregator based elements always started at running time 0,
now it's possible to select the first input buffer running time or
explicitly set a start-time value.

Created attachment 308344 [details] [review]
compositor: Add unit tests for the new aggregator start-time-selection property

Any objections for getting this merged before 1.6?

No objections from me. I'd rename the 'set' one to 'custom' or 'property' or so though </bikeshed>