Created attachment 256101 [details] [review]
Fix so that matroskamux updates header

This is a hack...don't know what the real fix might be...works though.

I'm not sure the format really supports adding/removing streams "in the middle". I think it should error out or return NULL if new pads are requested after streaming has started.

Yes, Matroska does not support that. It should error out if someone tries to do that.

I am using matroskamux with "streamable=true" and I need it to be able to request pads in this manner.  The patch, although a hack, allows this to occur.  So, none of the GStreamer multiplexers allow pads to be requested "on-the-fly"?

(In reply to comment #2)
> I'm not sure the format really supports adding/removing streams "in the
> middle". I think it should error out or return NULL if new pads are requested
> after streaming has started.

I could see this if one was putting content to a file...however, if one is streaming data over an Ethernet connection then it seems that the capability of "on-the-fly" pad request and subsequent header updates to reflect change in available tracks would be a necessity.

(In reply to comment #5)
> (In reply to comment #2)
> > I'm not sure the format really supports adding/removing streams "in the
> > middle". I think it should error out or return NULL if new pads are requested
> > after streaming has started.
> 
> I could see this if one was putting content to a file...however, if one is
> streaming data over an Ethernet connection then it seems that the capability of
> "on-the-fly" pad request and subsequent header updates to reflect change in
> available tracks would be a necessity.

If streamable=false then don't allow "on-the-fly" request pads and if streamable=true then allow them with header updates.

http://matroska.org/technical/streaming/index.html

Note that this says that for streaming you can use multiple segments, not that you just duplicate some of the headers. So whenever the configuration changes you could, similar to HLS/DASH/etc, start a new Matroska stream and send that without interruption to the client.

I am sorry in advance if this is a stupid question.  :-)  Would the re-configuration on the application level entail the destruction of the current matroskamux pipeline and instantiation of a new matroskamux pipeline?

(In reply to comment #8)
> Note that this says that for streaming you can use multiple segments, not that
> you just duplicate some of the headers. So whenever the configuration changes
> you could, similar to HLS/DASH/etc, start a new Matroska stream and send that
> without interruption to the client.

Yes, for example

With all do respect, seems like it accomplishes the same thing as the patch, but with more overhead.  I fail to see the difference besides stopping the pipeline, re-requesting pads for previous mux stream(s) attached to matroskamux and requesting pad for new stream and then restarting the pipeline with reconfigured matroskamux.  For my own edification, would you be able to point me to the specification that states that the Matroska container does not support dynamically added streams?  :-)

Actually the matroska spec does not forbid to have multiple Tracks elements. So in theory that should be fine indeed, sending all headers new but not starting a new Segment. However I would be surprised if any software handles that correctly currently.

It should be safer to start a new Segment and then all the other elements again. So the question is if we want to have this complexity inside the muxer or put it on the application side. Your patch would not be enough btw, you would additionally need to make sure that the next segment starts with keyframes for all streams for example.

Totem w/ GStreamer 0.10 (Git) and GStreamer 1.2 (with unmodified matroskademux) plays as do mplayer and Windows Media Player with CCCP installed.  VLC does not and I am going to look in to that.

Yes, I understand that my patch is really, really crude...it is just a proof-of-concept.  ;-)

How does matroskamux plugin ensure that new segments start on a keyframe for a stream that is added before the pipeline is set to play and container playing commences?  Say the source coming in to the matroskamux plugin is live.  If h264parse plugin (for example) is upstream from matroskamux plugin, does it wait until it encounters a keyframe before pushing a buffer to the matroskamux plugin?  Sorry to be such a pest.  :-)

...(In reply to comment #15)
> How does matroskamux plugin ensure that new segments start on a keyframe for a
> stream that is added before the pipeline is set to play and container playing
> commences?  Say the source coming in to the matroskamux plugin is live.  If
> h264parse plugin (for example) is upstream from matroskamux plugin, does it
> wait until it encounters a keyframe before pushing a buffer to the matroskamux
> plugin?  Sorry to be such a pest.  :-)

meant to say "container building commences" not "container playing commences"...

At the start of a h264 stream there should be a keyframe. If it isn't you're probably doing something special already with dynamic pipelines and then it's your job :) Or you're having a stream that was cut, in which case it's also something you should ensure.

It's probably not strictly required for Matroska streams to start with keyframes, but without it could become painful to handle for applications... if a frame of the new stream for example references a frame of the old one. That's why I'm thinking it should be done at a higher level than matroskamux, like what is done in hlssink for example.

(Assume HTTP steaming here) If you have video #1 stream in a Matroska container streaming in to a device and it is happliy playing both and the source suddenly removes video #1 from the container and replaces it with video #2 stream (and, of course, reflects this via a new segment), then the device which is receiving it should see the new segment that video #1 has gone away and restart it's video decoder.  If the device decides to automatically start playing video #2 then the first thing the decoder is going to do is wait for a keyframe.  I agree leaving the decoder running during the video #1 stream to video #2 stream transition without ensuring that video #2 containes a keyframe could be detrimental and produce a terrible user experience visually.  However, if Matroska does not strictly require segments to begin on keyframes then this scenario would arise whether the dynamic addition of streams to a container is handled in the application or the matroskamux plugin.

Basically the easiest way of handling all this is to do it like it's done in hlssink, without increasing complexity inside the matroska muxer element.

Sure, I will take a look.  I use hlssink in another application but have never perused it's source.

So, I also looked at mpegtsmux and it seems to adhere to the same paradigm as matroskamux where it only builds the objects which denote number of streams, tracks, (what have you) only once when playing.  Any sink pads that are requested and acquired from it by the application while it is playing do not cause a new object (in MPEGTS PAT/PMT) to be rebuilt.  I guess this is a paradigm adhered to by all of the mux plugins?  It seems to me, if this is the paradigm, then it needs to be enforced by throwing an error when an application attempts to request a pad when the mux is playing.

Thanks for your help...I took out my hack from matroskamux and modified my application per your suggestion and all is working just fine.  Thanks for your help and sorry for crying "Wolf!" on this.  Looks like no mods need to be made to matroskamux at all.

No problem, I'm glad it works properly now :)