GNOME Bugzilla – Bug 711522
Need a standard way to get format-specific stream ID (for HTML5 track "id")
Last modified: 2018-11-03 12:19:12 UTC
I'm trying to implement the "id" attribute on HTML5 TextTrack, AudioTrack and VideoTrack objects: http://www.w3.org/TR/html5/embedded-content-0.html#dom-texttrack-id http://www.w3.org/TR/html5/embedded-content-0.html#dom-audiotrack-id Currently, what the "id" is isn't standardized, but the WG is starting soon and presumably it will be the standard ID for each format (PID for MPEG-TS, stream serial number for Ogg, track UUID for Matroska). Some options I considered: * Stream ID: This is nice because every stream has one, and it contains the value we want at least in Matroska. One problem is that the stream ID isn't currently defined to have any meaning besides being a unique identifier (although it could be if we wanted it to). The other problem is that all streams need a stream ID, but not all formats actually have an internal ID (the output of audiotestsrc, or WAV files for example). * GST_TAG_SERIAL: This is defined as an unsigned integer, and not all formats use an unsigned integer for their ID's (Matroska comes to mind). Even if all formats did now, it wouldn't be future-proof. My suggestion is to add a GST_TAG_ID, which is defined as "the ID for this stream/track in the original format", which could then contain the PID, stream serial, etc.
Another option is to make a series of tags for each format, like: GST_TAG_MATROSKA_TRACK_NUMBER GST_TAG_MATROSKA_TRACK_UID GST_TAG_MATROSKA_TRACK_TYPE etc. I think it would be better if we could handle multiple formats in a consistent way, but that's another option if it comes to that.
Related WebKit bug: https://bugs.webkit.org/show_bug.cgi?id=123901
Another problem with using the ID from the stream start even is that there's no way to figure out which part is the one you want. For example, here's a stream ID from a video track in WebKit: 9f7fea11efe01e5a6961dd297cbd7f77ddb568657cb2fad96783878d99416edd/00000050 Ok, so the PID is 50 (but it's we need to remove the leading 0's). And here's a text track: 9f7fea11efe01e5a6961dd297cbd7f77ddb568657cb2fad96783878d99416edd/00000050/user_data_src/src I doubt we could determine the PID here, even though a human can easily pick it out.
Created attachment 259216 [details] [review] Add GST_TAG_TRACK_ID I think tags are likely the best way to go. I'm attaching example patches.
Created attachment 259217 [details] [review] Add GST_TAG_TRACK_ID for Matroska (using TrackUID)
Created attachment 259218 [details] [review] Add GST_TAG_TRACK_ID for MPEG-TS (using PID)
I would probably call that TRACK_CONTAINER_ID. That would make it more explicit what this is, and differentiate it from the stream-id. Open issues: what to do for containers in containers?
Comment on attachment 259216 [details] [review] Add GST_TAG_TRACK_ID Rename and "Since: 1.4". Also before merging that someone else should agree to this API too
Comment on attachment 259217 [details] [review] Add GST_TAG_TRACK_ID for Matroska (using TrackUID) Why as an integer and not a hexadecimal string?
Comment on attachment 259218 [details] [review] Add GST_TAG_TRACK_ID for MPEG-TS (using PID) Same question here
I used uint64 since that's how we store them: (In matroska-ids.h) struct _GstMatroskaTrackContext { ... guint64 uid, num; ... }; Now that I think about it, I'm not sure if that's correct, since the spec doesn't give any particular length: http://matroska.org/technical/specs/index.html#TrackUID I'm going to ask about this in the "In-band tracks community group": http://www.w3.org/community/inbandtracks/
My point was more that having it as a integer seems like a rather arbitrary choice. You could as well do it as an hexadecimal string, or zero-padded integer, or ... :) What are you going to do with file formats where the tracks are identified by something else, like in MXF where it is an 128 bit UUID. How is this used in the W3C spec? Does it have to be of a specific format, is it referenced from outside the media player somehow, e.g. by some JavaScript that wants to select a track by an ID hardcoded somewhere?
The main way this can be used in HTML is that you can pick a track in the URI, something like this to automatically select tracks 5 and 10: <video src="myvideo.webm#track=5&track=10"/> http://www.w3.org/TR/2011/CR-media-frags-20111201/#naming-track It probably shouldn't be arbitrary though.
Btw, regarding comment #3, I just wanted to say that nothing prevents you from putting additional things into a stream-id string, such as /foo/bar/pid=50,tid=abcdef9a/0005/bleh just to give an example. If it's nice is a different question of course.
Here's my email to the WG: http://lists.w3.org/Archives/Public/public-inbandtracks/2013Nov/0023.html I'll update this when I have a better idea of how to format the ID's.
Created attachment 259680 [details] [review] Add GST_TAG_TRACK_CONTAINER_ID Changed the name to GST_TAG_TRACK_CONTAINER_ID as requested.
Created attachment 259681 [details] [review] Add GST_TAG_TRACK_CONTAINER_ID for Matroska (TrackUID) The inband track WG agreed that decimal was the best representation, at least for these two formats.
Created attachment 259682 [details] [review] Add GST_TAG_TRACK_CONTAINER_ID for MPEG-TS (PID)
Hm so GStreamer's TrackUID doesn't match mkvinfo's :\ mkvinfo is using libmatroska, but I don't see any reason why they would disagree, since both are correctly handling it as a uint64.
Hm I think it might just be this file: https://svn.webkit.org/repository/webkit/trunk/LayoutTests/media/content/counting-subtitled-srt.mkv mkvinfo says (lots of superfluous info cut out): |+ Segment tracks | + A track | + Track number: 1 (track ID for mkvmerge & mkvextract: 0) | + Track UID: 2403036717 | + Track type: video | + A track | + Track number: 2 (track ID for mkvmerge & mkvextract: 1) | + Track UID: 1069831963 | + Track type: subtitles | + A track | + Track number: 3 (track ID for mkvmerge & mkvextract: 2) | + Track UID: 648149798 | + Track type: subtitles | + Language: fre |+ EbmlVoid (size: 1122) But GStreamer gives: Tags: taglist, container-format=(string)Matroska, video-codec=(string)"DivX\ MPEG-4\ Version\ 4", track-container-id=(string)565333916076699181, minimum-bitrate=(uint)23600, bitrate=(uint)161363, maximum-bitrate=(uint)312800; Tags: taglist, track-container-id=(string)3532951484994180891, language-code=(string)en, container-format=(string)Matroska; Tags: taglist, track-container-id=(string)8676205028037491494, language-code=(string)fr, container-format=(string)Matroska;
Hm mkvalidator says the file appears to be valid.
Ok, so the TrackUID in GStreamer is correct. mkvinfo is wrong (https://trac.bunkus.org/ticket/935).
There's actually som disagreement about using the PID for this in MPEG-TS. I think TrackUID is definitely the best for Matroska, but I think it might be good to talk to someone involved in WebM before deciding that.
Here's a complication: For Ogg, we'll want to use the Skeleton "Name" header if we can, but not all Ogg files have a Skeleton section, so we'd probably want to fallback to the stream serial number in that case. I'm not sure how complicated it would be to implement that as a tag.
I'm wondering if it might be better to just provide all of the headers or track info as tags. For example: GST_TAG_MATROSKA_TRACKUID GST_TAG_MATROSKA_TRACKNUMBER GST_TAG_MATROSKA_NAME GST_TAG_MATROSKA_LANGUAGE ... GST_TAG_OGG_NAME GST_TAG_OGG_TITLE GST_TAG_OGG_ROLE GST_TAG_OGG_STREAM_SERIAL_NUMBER ... GST_TAG_MPEGTS_PID Then the browser can figure out what it wants to do with it. It might be nice if we could provide this in a general way, since Ogg Skeleton allows free-form headers. Maybe just GST_TAG_OGG_SKELETON_HEADERS, with a list of headers?
That could be done, even just be implemented inside the demuxers (you can register new tags there too). It just seems a bit inconvenient for the browser to implement all these. But I guess that can't be prevented because the spec here seems very open and not well-defined.
Chromium plans to implement this through a new FFMPEG "id" metadata, so it seems like adding a GST_TAG_TRACK_ID should work.
What is it supposed to contain? Is that defined for each relevant container format?
There will be one for each container format. So far the plan is TrackUID for Matroska/WebM, PID for MPEG-TS (although one group wants a descriptor to be used for this), and Ogg is the tricky one since there could be several different IDs, but everyone seems to agree that the skeleton Name header should be used if it exists. You can see the work-in-progress here: https://wiki.xiph.org/SkeletonHeaders#Name Chromium already uses TrackUID for WebM, and since they basically own that spec I'd be surprised if that changed. https://code.google.com/p/chromium/issues/detail?id=313601
For Ogg Skeleton, the HTML5 spec says specifically that we should use the Name header: http://www.w3.org/html/wg/drafts/html/CR/embedded-content-0.html#dom-audiotrack-id I'll upload a gst-plugins-base patch to demonstrate (similar to the "kind" patch).
Created attachment 269728 [details] [review] Add GST_TAG_TRACK_ID This adds GST_TAG_TRACK_ID, similarly to GST_TAG_TRACK_KIND.
Created attachment 269729 [details] [review] Use Ogg Name header for track id This uses the Ogg Skeleton Name header for track id, as specified by the HTML5 spec: http://www.w3.org/html/wg/drafts/html/CR/embedded-content-0.html#dom-audiotrack-id If the kind tag patch for Ogg is accepted, this patch can be made much smaller, since most of it is the setup to read Ogg Skeleton headers.
It looks like we'll be using TrackNumber for WebM, because it's guaranteed to always be there, while TrackUID isn't (TrackUID is supposed to be mandatory, but videos can't even play without TrackNumber). https://github.com/silviapfeiffer/HTMLSourcingInbandTracks/commit/aadb4a59d05ade02abcae9cf842714ec7c9f91b2 http://lists.w3.org/Archives/Public/public-inbandtracks/2014May/0001.html
The W3C has a spec that covers all aspects of track attributes now: http://rawgit.com/w3c/HTMLSourcingInbandTracks/master/index.html Is it possible to get this in?
Maybe this should go to gst-plugins-base/libgsttag and be named GST_TAG_HTML_TRACK_UID, but otherwise I think this can go in if it's properly specified in the spec.
(In reply to Sebastian Dröge (slomo) from comment #35) > Maybe this should go to gst-plugins-base/libgsttag and be named > GST_TAG_HTML_TRACK_UID, but otherwise I think this can go in if it's > properly specified in the spec. This isn't necessarily a UID. It could be an MPEG-TS PID, or a caption channel number, or a MPEG-4 track ID, etc.
As this was requested again recently, maybe we should just go for it? Also needed for MP4 and WebM. GST_TAG_CONTAINER_SPECIFIC_TRACK_ID? GST_TAG_HTML_TRACK_ID? Any other variant? :)
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/45.