GNOME Bugzilla – Bug 729134
mpegtssection: Include a timestamp
Last modified: 2018-11-03 13:23:02 UTC
I'm looking at exposing MPEG-TS sections as metadata in HTML5, and it would be nice if we had consistent timing information. GST_MESSAGE_TIMESTAMP() should give us a time, but it's always invalid for MPEG-TS section messages. The only timestamp I see is MpegTSPacketizer2.last_in_time. Would that be a reasonable time to use for the message?
Hm except last_in_time seems to be invalid too.
Hi Brendan. Do you have a code snippet I could use to reproduce this?
Created attachment 275387 [details] Example code to print MpegTsSection message timestamps Here's some example code. You can run it against this clip: https://svn.webkit.org/repository/webkit/trunk/LayoutTests/media/content/user-private-streams.ts
Created attachment 275389 [details] Same example, but in Python Tim helped me finish the Python version, which might be a little easier to run. It should just be: gst-git ./mpeg-ts-section-timestamp.py https://svn.webkit.org/repository/webkit/trunk/LayoutTests/media/content/user-private-streams.ts
Cool! Thanks a lot Brendan. I will look into this tomorrow.
Biggest question would be what time you want to have for this. Should it be based on the PCR, if any? In that case there is code in the packetizer to get this IIRC.
The PCR seems reasonable. By "if any", do you mean that there may not always be one? From the articles I can find, it sounds like the PTS depends on the PCR, so it seems like there should always be on, but maybe I'm missing something.
Brendan, Do you mind giving us the link to those articles? Curious.
http://en.wikipedia.org/wiki/MPEG_transport_stream#PCR > To enable a decoder to present synchronized content, such as audio tracks matching the associated video, at least once each 100 ms a Program Clock Reference, or PCR is transmitted in the adaptation field of an MPEG-2 transport stream packet. [...] Timing in MPEG2 references this clock. For example, the presentation time stamp (PTS) is intended to be relative to the PCR. I guess according to this, a PCR isn't required: http://en.wikipedia.org/wiki/Program-specific_information#PMT_.28Program_map_specific_data.29 > The packet identifier that contains the program clock reference used to improve the random access accuracy of the stream's timing that is derived from the program timestamp. If this is unused. then it is set to 0x1FFF (all bits on). I'm not sure how we would provide any reasonable timestamp without the PCR though. Any ideas? We could just leave it at GST_CLOCK_TIME_NONE if we have no clock..
Would it be possible to use the PTS here?
A lot of sections won't have an associated PCR PID (essentially all the non-program specific ones). What is the bigger picture here ? What do you need timestamps for ?
We're trying to expose the metadata to HTML, and want some sort of timestamp since some things are time-sensitive (mostly "user private data" I think).
Some (dsm-cc/carousel) sections will contain timestamps *in* them, for events that do indeed need to take place at a given time. Those could be re-used (converted to running time). Putting generic timestamp is tricky, but I guess you *could* put the current PCR time, converted to running time (otherwise you have no time reference to sync against). Note that this last PCR time will be slightly later than equivalent A/V frames due to the whole synchronization model. That being said, you're left with one issue ... how to know when that running time is (and have it paused when it should be paused, etc...). Getting running-time only solves one part of the synchronization equation, you'd also need the current clock, current pipeline base-time and whether the pipeline is running or not. Two ways forward: 1) You post selected sections as custom in-band events downstream in the pipeline and get them posted on the bus when the running-time is reached. The problem with this is that it delays your section processing. 2) You get the clock/base-time, track the pipeline state yourself and use the various gst_clock_*wait* API to know when a certain action should be triggered. This way you can do all your processing, decide which actions you want to wait against, etc... For the second option, it might be nice to have a convenience object to track all of this. Some kind of "user" out-of-band sink for any kind of timed action.
I'm not sure I understand the details of MPEG-TS enough to respond to this. I'm going to try talking to some coworkers to see what would be best.
Why can't we just put this information in the message TIMESTAMP? http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstMessage.html#GST-MESSAGE-TIMESTAMP:CAPS Presumably the "timestamp when the message was created" would be the PCR, unless this has a different meaning than I'm expecting? It would be fine if we saw the message before that time. For ad-insertion, it's actually useful to see the event before we're supposed to handle it, since that gives us time to pre-load the ad. Am I just re-describing option 2?
Not sure we could re-use that field, since it's when the message "was created", whereas you want to know when "the content of this message/event applies". So assuming we go for option 2, which would be the sanest, there's yet-another-option which is easier to handle app-side. Depending on how granular/reactive you want to be, this could be acceptable. We could use (PCR or PTS converted to) stream time for those time field. And then it would be up to your app to regularly (say 2 times per second) query the current pipeline position (which is in stream time), and you could apply/handle those messages/events/actions when you reach that stream time. That being said, I'm not a great fan of calculating/setting that field on *all* mpeg-ts sections. One reason is performance, the other one is just that ... it makes no sense for quite a few sections (PAT, PMT, EIT, ... aren't timed events). Can you provide a list of specific sections you know you'll need timestamps ?
We're using WebKit's timeline callbacks, so we don't actually need any callback support, just accurate times. The PMT and PAT actually are timed events because there can be multiple in a single stream and they apply to everything after them. Normally I wouldn't be concerned about accuracy with this, but for HTML work, we need to be consistent with other implementations. It's sounding like so far the PMT and private sections are the only ones we plan to expose as timed events.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/144.