What's the purpose/rational/use case of segments in matroska?

This is unlikely to ever get implemented (anywhere) without sample files.

Maybe the matroska folks could whip up a sample file and put it into their sample downloads section?

have you seen here?
http://www.matroska.org/samples/matrix/index.html

It's the same as chained ogg files as far as I understand it... so if a new segment comes after the first, re-init the demuxer, add new pads, etc.

Will look at this in the next time...

is there any progress with this bug?

*** Bug 644808 has been marked as a duplicate of this bug. ***

Bug #644808 had a start of a patch but needed some more work.

I have also seen this issue with WebM fragments in MSE support of WebKitForWayland.

matroskademux receives the WebM elements in this order and in this state:

Element id GST_EBML_ID_HEADER, state GST_MATROSKA_READ_STATE_START 
Element id GST_MATROSKA_ID_SEGMENT, state GST_MATROSKA_READ_STATE_SEGMENT 
Element id GST_MATROSKA_ID_SEEKHEAD, state GST_MATROSKA_READ_STATE_HEADER 
Element id GST_EBML_ID_VOID, state GST_MATROSKA_READ_STATE_HEADER 
Element id GST_MATROSKA_ID_SEGMENTINFO, state GST_MATROSKA_READ_STATE_HEADER 
Element id GST_MATROSKA_ID_TRACKS, state GST_MATROSKA_READ_STATE_HEADER 
Element id GST_MATROSKA_ID_CUES, state GST_MATROSKA_READ_STATE_HEADER 


Element id GST_MATROSKA_ID_CLUSTER, state GST_MATROSKA_READ_STATE_HEADER 
Element id GST_MATROSKA_ID_CLUSTERTIMECODE, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_SIMPLEBLOCK, state GST_MATROSKA_READ_STATE_DATA 
*
*
*
Element id GST_MATROSKA_ID_CLUSTER, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_CLUSTERTIMECODE, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_SIMPLEBLOCK, state GST_MATROSKA_READ_STATE_DATA 
*
*
*
Element id GST_EBML_ID_HEADER, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_SEGMENT, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_SEEKHEAD, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_EBML_ID_VOID, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_SEGMENTINFO, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_TRACKS, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_CUES, state GST_MATROSKA_READ_STATE_DATA


Element id GST_MATROSKA_ID_CLUSTER, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_CLUSTERTIMECODE, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_SIMPLEBLOCK, state GST_MATROSKA_READ_STATE_DATA 
*
*
*
Element id GST_MATROSKA_ID_CLUSTER, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_CLUSTERTIMECODE, state GST_MATROSKA_READ_STATE_DATA 
Element id GST_MATROSKA_ID_SIMPLEBLOCK, state GST_MATROSKA_READ_STATE_DATA 
*
*
*

MatroskDemux ignores the element "GST_EBML_ID_HEADER" in
"GST_MATROSKA_READ_STATE_DATA" state, so it doesn't reset the streams parser
and doesn't parse the new EBML Header.

What would have to happen is that the whole state is reset when a new EBML header is parsed, but I'm not sure what additionally needs to be done. Ideally seeking between the segments should also be possible, timestamps (well, running-time/stream-time but PTS/DTS not) will have to continue with the new fragment instead of starting from 0, .... It might make sense to check how qtdemux handles this with fragmented MP4.

Created attachment 343228 [details] [review]
support the multi-segment matrosk/webm in MSE

I tested this patch, and it work fine.
what do you think ?

Does this work with a file created by concatenating two existing Matroska files? There's probably some more work needed here for timestamps at least, and as said before for seeking between segments.

And streams/caps can change, right? So we need some logic to add/remove pads and all that, no?

I agree with you in the case of the matroska file, but in MSE fragments context i am not sure it is necessary to do all these modifications

like a first step i think it is sufficient for MSE context, no ?

Here is context in detail:

- we use webkitforwayland as browser engine.
- we play youtube with MSE (Media Source Extension) 
- We use the webm container with vp9 codec
see this specifications: 
https://www.webmproject.org/docs/container/
https://www.w3.org/TR/media-source/
https://www.w3.org/TR/mse-byte-stream-format-webm/
- We use the matroskademux to demuxe the webm/video stream into video vp9 


I will try see how is treated in qtdemux with mp4

(In reply to y.bandou from comment #12)
> I agree with you in the case of the matroska file, but in MSE fragments
> context i am not sure it is necessary to do all these modifications
> 
> like a first step i think it is sufficient for MSE context, no ?

It might be sufficient for your very use case, but it's incomplete from a GStreamer point of view and will cause confusing problems (instead of clear errors) for all other cases then your application.

Created attachment 362212 [details] [review]
Alternative version of the patch. Honors the DISCONT flag to produce the right GstSegments.

I found the issue described on this bug last week while analyzing why YouTube 2018[1] test "20. VideoBufferSize" failed for us in WebKitGTK+.

What the test does in our case is to append (to process) the same 1MB video file[2] one time after another. It expects to get [0, 1] PTSs each time and uses SourceBuffer.offset to convert that into [0, 1][1, 2][2, 3][etc.]

That file starts with a GST_EBML_ID_HEADER which is expected in the GST_MATROSKA_READ_STATE_START stage but can't be parsed while in GST_MATROSKA_READ_STATE_DATA. In fact, the demuxer never goes out of that DATA state.

The solution I finally got during the GStreamer Hackfest was very similar to Yacine's. I just found this bug later when Yacine told me about it the days after, during the Gstreamer Conference. X-D

One additional issue I found was that, even if we "convince" matroskademux to expect GST_EBML_ID_HEADER in the DATA state so it goes on and outputs the buffers, the demuxer will also generate GstSegments starting from the last output PTS. This means that the newly generated buffers (starting from 0 in my test case) will be out of the GstSegment and thus will be ignored by the decoder and/or the sink.

I asked Sebastian about that issue and how similar it is to the TFDT issue in qtdemux[3]. He suggested to modify our app (WebKit) to set the DISCONT flag on the appended buffers and to honor that flag in matroska demux and reset the segment position. This basically works for me with my specific test case.

I acknowledge that my test case has some particularities that somehow makes it special and not generalizable:

- The number of streams and its caps remains constant (as it's the same data). According to the pipeline dumps, matroskademux doesn't seem to create extra pads and just reuses the existing ones.
- All the chunks start from 0. I don't know what would happen if the data in a new append wanted to generate eg: [5, 6] instead of always [0, 1]. I guess I'll find out when I focus on the next tests.

Another thing that I don't understand completely is why my patch works (for me) even though what I do to prepare the transition to GST_MATROSKA_READ_STATE_SEGMENT is simpler than what Yacine does.


[1] http://yt-dash-mse-test.commondatastorage.googleapis.com/unit-tests/2018.html
[2] http://yt-dash-mse-test.commondatastorage.googleapis.com/unit-tests/media/vp9-video-1mb.webm
[3] https://bugzilla.gnome.org/show_bug.cgi?id=754230

(The explanations about Matroska can be a bit confusing, so I'm adding a diagram.)

It seems that the discussion of this issue has geared towards MSE use cases that actually don't have a big resemblance to the original issue.

The original issue was about support of multiple Matroska segments in a single file. This is something that Matroska used to allow and some players (e.g. VLC, mpv) still support.

A Matroska Segment is a the top level object in Matroska containing both metadata (tracks definition) and video data. Usually a .mkv file contains exactly one segment, but this is not the case for multi-segment files. "Matroska File Format" (2009) claims:

> There can be several SEGMENTs in one MATROSKA file, but this is not encouraged to be done, as not many tools are able to handle multisegment MATROSKA files correctly.

Nowadays the Matroska spec claims:

> A Matroska file is composed of 1 Segment.

Multisegment Matroska files are created by concatenating (e.g. with `cat`) two or more Matroska files. Although technically you can concat any two Matroska files, the use cases where in currently available players this can produce a useful result other than just playing the first segment when that file is opened are two:

* Matroska files using ordered chapters with external references pointing to another segment. (these are usually used to save space by extracting the opening and credits of a series to separate files and linking them in each episode file instead of copying them again and again). Support for this is far from universal, but not unheard of in desktop players (VLC, mpv, MPC).

* Linked segments. Segments can form a linked list (see PrevUID and NextUID). A movie (or SegmentFamily in Matroska parlance) is intended to be played as the concatenation of the timelines of each of its segments, e.g. if the first segment is 1 hour long, 0:00:00 from the second segment becomes 1:00:00 in the timeline of the movie shown in the UI. Support for this feature is pretty rare, as ordered chapters with external references already allow to obtain the same user-visible behavior and more. Nevertheless, MPC and VLC support it, though VLC crashes very easily with this feature (in my testing once in every ~four seeks).

Such segments can be created with readily available free multiplatform software like MKVToolNix GUI.

Both of these features require a mechanism for finding a Segment given a UID. Players usually scan the directory where the file is stored and parse every .mkv file to construct a table of Segment UID -> (file, offset) [offset is only needed if multi-segment files are supported]. This process is relatively fast if the number of files in the folder is reasonable as usually only the few first few bytes per file need to be read.

`file` needs not to be a different file. Indeed, that seems to be the case OP originally had. We will refer to such files as Multisegment Matroska Movie Files (a single file containing a single movie containing several Matroska segments).

From a practical perspective, the usefulness of Multisegment Matroska Movie Files is quite limited: if you wanted to save space by using ordered chapters with external references, it makes sense that those references are stored separately; if you wanted to split a very long Matroska movie in several segments (e.g. to accomodate file system maximum file size restrictions) it makes sense that they are in separate files. Multisegment Matroska Movie files are a little more than a simple way to "undo" that separation and put them together in the same file without remuxing.

It's possible also to concatenate completely unrelated, unlinked Matroska segments in the same file. Currently available players will ignore all but the first segment, using the rest just as support for finding references.

Neither ordered chapters nor linked segments are supported by matroskademux currently.

In fact, they cannot be implemented with typical source elements such as filesrc. For a source element to work with a hypothetical matroskademux implementation supporting any of these features, such source would need to support a request for locating and loading a given Segment UID, and in practice this would mean that it would also need a scan mechanism like described above.

A new source element would be required, let's call it `matroskafinder`, that similarly to filesrc and others supported a location property, but also accepted "Segment UID" requests, scanned the folder containing the original file and fed the correct file to matroskademux. Ideally, this would be a configurable bin: a factory function for instantiating the inner source elements could be provided, so that you could play ordered chapters over the network and the scan mechanism would be replaceable e.g. to query a database instead of downloading each file in certain applications.

Another implementation challenge is that both of these features require tracking new timeline mappings. As far as I know, this would be all the timeline levels, from raw to user facing (none of the names of the timelines are official):

1. Block Timecode (coded timecode in the frames, relative to the starting time of its container Cluster).
2. *Cluster time (Block Timecode + parent Cluster start Timecode).
3. CodecDelay adjusted track time (Cluster time - track's CodecDelay, if any).
4. Offset track time (CodecDelay adjusted track time + TrackOffset, deprecated).
5. *Segment time (offset track time after applying an edition entry, i.e. an ordered set of chapters, see the definition here: https://www.matroska.org/technical/specs/index.html). In the case of external ordered chapters, edits are recursive, as their time codes are in segment time of the linked segment.
6. *Movie time (Segment time + end time of the previous linked Segment).

The timelines marked with a star are parenting, i.e. it is formed by assembling together several instances of an immediately lower level timeline class.

Making things even harder, segments are not guaranteed to have the same kind and number of tracks, codecs, defaults, etc. (though they often do), so it may be necessary to reset most of matroskademux, including its pads, when switching to a different segment (including when reading an external ordered chapter). Constructing the movie timeline map may be tricky.

How could Multisegment Matroska Movie File support like OP wanted be implemented?

As a prerequisite, support for the other features explained here -- at least ordered chapters with external references, would make a lot of sense. Ordered chapters with external references to other files are much more common than concatenated Matroska files of any kind.

With that done, the only addition would be ensuring that the Segment scanner actually tries to find and several Segments per file (by reading past the last element specified in the Meta Seek, looking for a Segment element) and that file sources can be requested to the factory in matroskafinder with a given offset and size (the ones covering the found segment).

The scanner would also search in the same file that is being currently played, finding the segments. This is the way this works in mpv. It's actually the same as if the segment was in another file.

Of course this all only is possible when matroskademux is the driving force of the pipeline, able to switch between different file sources.

That all is a daunting task so it should be no surprise that support for movies with multiple Matroska segments is so spotty.


Next topic: What about "MSE segments"?

MSE has the concepts of "initialization segment" and "media segment", whose definitions are specific to MSE and depend on the container format. In the case of WebM a "MSE media segment" is a Cluster element and a "MSE initialization segment" is the portion of a WebM file that defines the header before clusters can appear.

MSE segments may appear in the stream in almost any order, with some limitations: the first ever MSE segment must be an initialization segment.

Further MSE initialization segments in the stream are limited to have the same number and type of tracks, allowing very little variation. An noticeable exception is codec data (e.g. PPS and SPS for MP4/h264). Usually applications insert a second initialization segment after a quality change.

Support of multiple MSE WebM segments in matroskademux revolves about the following question:

  When working in push mode, what should the demuxer do if it finds a new Matroska header?

Note the difference with the Multisegment Matroska Movie playback problem exposed above. In the above case, the demuxer searched everywhere (including in the same file, past the current Matroska Segment boundary) for an specific Matroska segment with a given SegmentUID. Here, it's the other way around. The demuxer was happy parsing the current Matroska Segment when suddenly it ended and a new one took its place. What should happen in this case?

Well, that's unrelated to the original problem in this bug so... To be continued in https://bugzilla.gnome.org/show_bug.cgi?id=793333

Let's keep this bug restricted to the original issue, even if it's not met with the same interest.

Created attachment 368191 [details]
Diagram of Matroska boxes for Segments and Chapters

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/2.