It's generally not necessary to send a new caps event when streams change if the caps do not change, same goes for getting a new buffer pool. The old one should continue to work as the stream format did not change.

Independent of that, requesting a new buffer pool would happen with the ALLOCATION query and has nothing to do with the caps event.

Why exactly is it necessary in your case, what's the use case?

My use case is stream has 4 video track with different resolution. When application set another video track active. input selector will send RECONFIGURE event to video decoder to old and new pad. As video sink is link to different video track and video decoder haven't send caps to video sink, video sink will proposal old buffer pool to the new active video decoder. It will cause issue.

So the caps for the different video streams are different? Then input-selector should forward the new caps downstream when switching, and also the video decoder will have sent an ALLOCATION query with the correct caps for every single stream to the sink, where it then retrieved its buffer pool.

It sounds to me like the problem lies somewhere in input-selector or in how your sink handles buffer pools.

input-selector just send RECONFIGURE event to video sink pad. Do you think input-selector should also responsible set active video caps to video sink? video sink only check if ALLOCATION query caps is same with video sink pad caps. Do you think video sink should reset pad caps based on ALLOCATION query caps? Which is the right way?

input-selector should send the new caps of the newly selected stream downstream before any serialized event/query or buffer. Then the sink will get it and know what caps to use for the following buffers. (It looks like it currently only does that for buffers, not serialized events/queries... which would mean that you could get the ALLOCATION query from the new video decoder before receiving the caps of that in your sink, but that shouldn't be a problem because see below).

Your sink should always provide a buffer pool for the caps that are specified in the ALLOCATION query. This should be completely independent of the caps you receive with the CAPS event. The CAPS event tells you what CAPS the buffers that follow it should be interpreted with, the caps in the ALLOCATION query tell you for which caps buffers should be allocated.
When you receive buffers in the sink you will be able to a) correlate buffer pool to buffer and b) received the CAPS event for the stream of that specific buffer some time before the buffer.

Many thanks for your explain.

I can understand video sink should prepare buffer pool base on ALLOCATION query. But video sink still need CAPS event to interpreted the input buffer. Which element should send the event or how video sink correlate buffer pool and input buffer?

The CAPS event is sent by the decoder, and before the first buffer input-selector will forward it when the stream is selected. Look for the call to gst_pad_sticky_events_foreach() in the input-selector code.


I assume you have a custom allocator and memory type in the buffers returned by your buffer pool. In that case you can store whatever is needed in that memory to make it usable by the sink together with the currently configured caps on the sink.

So, the video track selection will implement like below:
Video decoder will send ALLOCATION query with caps when negotiation.
Video sink should propose video buffer based on query caps.
input-selector will send all sticky event (SEGMENT, CAPS...) to video sink before push first buffer.

Video buffer pool only based on ALLOCATION query caps. CAPS event is used to interpreted video sink input buffer. Video sink also need check input buffer meta data to interpreted video buffer.

Is my understand right?

Yes, that sounds correct. If it is not like that you found a bug somewhere :)

Note however that the decoders for *all* 4 video streams will do an ALLOCATION query in the very beginning. So the sink will have to provide 4 different buffer pools, which might not be ideal depending on your sink. When selecting a new stream, a RECONFIGURE event is going to be sent to the newly selected decoder, and it will do another ALLOCATION query to see if something has changed.

If there's anything that can be changed in GStreamer to make your life easier, or if anything is not clear yet, please let us know :)

Thank you again.

I have test above method and it works. So the issue can be resolved without my patch.

I found one thing, video sink will deactive preview buffer pool and free all video buffer memory even if the video tracks has same caps. It seems must do it as video decoder maybe change video buffer pool configure, so video buffer pool need in inactive state. Do GStreamer has any method to avoid video sink reallocate memory when caps is same?

(In reply to comment #10)
> Yes, that sounds correct. If it is not like that you found a bug somewhere :)
> 
> Note however that the decoders for *all* 4 video streams will do an ALLOCATION
> query in the very beginning. So the sink will have to provide 4 different
> buffer pools, which might not be ideal depending on your sink. When selecting a
> new stream, a RECONFIGURE event is going to be sent to the newly selected
> decoder, and it will do another ALLOCATION query to see if something has
> changed.

Video sink only received the input-selector's active pad's ALLOCATION query. Only need provide one buffer pool. input-selector will send RECONFIGURE event to old active and new active pad and new active pad ALLOCATION query will reach video sink.

I wonder why implement track selection after audio/video decoder? Is it possible to implement in demuxer?

It's possible but not implemented yet. This is related to this bug #646638. We need some generic API to do stream deactivation/activation and stream selection inside the pipeline.