GNOME Bugzilla – Bug 596321
qtdemux: add support for fragmented MP4 and "mfra" boxes
Last modified: 2010-12-03 16:03:06 UTC
See: http://blog.cmstream.net/2009/07/fragmented-mp4-and-smooth-streaming.html and sample content from: http://www.microsoft.com/downloads/details.aspx?FamilyID=e44b0a2d-2e0c-48ff-bf57-3d05a20e2f6a&displaylang=en
fyi hopefully, I can post something soon.
Created attachment 156472 [details] [review] qtdemux: traverse the node passed as parameter Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156473 [details] [review] qtdemux: add fragmented mp4 fourcc's Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156474 [details] [review] qtdemux: Add mp4 node types Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156475 [details] [review] qtdemux: Add helper to parse atom's uint from 1 to 4 bytes Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156476 [details] [review] qtdemux: Add dump for mfro atoms Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156477 [details] [review] qtdemux: Add dump for tfra atoms Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156478 [details] [review] qtdemux: Add dump for tfhd atoms Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156479 [details] [review] qtdemux: Add dump for trun atoms
Created attachment 156480 [details] [review] qtdemux: Add dump for sdtp atoms
Created attachment 156481 [details] [review] qtdemux: Add dump for trex node
Created attachment 156482 [details] [review] qtdemux: Add dump for mehd node
Created attachment 156483 [details] [review] qtdemux: Check in the header loop if it's a fragmented format Based on patch by: Marc-André Lureau <mlureau@flumotion.com>
Created attachment 156484 [details] [review] qtdemux: parse the track id from the track header Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156485 [details] [review] qtdemux: parse fmp4 samples information The fragmented mp4 format stores the tracks and samples information in the 'moof' boxes, which are appended before each fragment (fragment->'moof'+'mdat'). The 'mfra' box stores the offset of each 'moof' box and their presentation time. The location of this box can be retrieved from the 'mfro' box, which is located at the end of the file. The 'mfra' box is parsed to get the offset of each 'moof' box and their presentation time. Each 'moof' box can contain information for one or more tracks inside 'tfhd' boxes. For each track in a 'moof', we have a 'trun' box, which contains information of each sample (offset and duration) used to build the samples table. Based on patch by Marc-André Lureau <mlureau@flumotion.com>
Created attachment 156486 [details] [review] qtdemux: allow pulling atoms with unknown size Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 156487 [details] [review] qtdemux: skip short tracks hack if it's fmp4 Signed-off-by: Marc-André Lureau <mlureau@flumotion.com>
Created attachment 156488 [details] [review] qtdemux: Do not parse samples if it's a fragmented format Samples have already been parsed in qtdemux_parse_trak().
Created attachment 156489 [details] [review] qtdemux: get the stream duration from the last sample In some fragmented files the track duration is wrong, because it uses the container's timescale instead of the track's one.
Theses patches add support for fragmented mp4 in pull mode. To create a fragmented mp4 file from an mp4 file fro testing, you can use the mp4split tool (http://smoothstreaming.code-shop.com/trac/wiki/Smooth-Streaming-Encoding)
I've only looked at the first few patches, but maybe you could squash these patches strategically? Surely the first one-liner can be squashes with some other commit where this change is needed? All the different dump functions should be squashed with the commits that add the new defines and table entries imho (and the last two should go together). There is no point in having all these as individual commits IMHO.
I split them to make the review easier. As you said most of them can be squashed, like those for the atoms dump, and those ones that add minor fixes for fmp4 files can also be squashed with the big one that parses the samples information.
Created attachment 156545 [details] [review] typefind-associate-fragmented-mp4-files-to-qtdemux Added patch to associate fmp4 files to qtdemux
Created attachment 157395 [details] [review] qtdemux-initialize-samples-with-pts_offset-0-to-avoi.patch If the pts_offset is not initialized to 0 it can have random values.
> I split them to make the review easier. I think reviewing these patches would be easier if you squashed them strategically as suggested. Wouldn't have asked for it otherwise :)
I spent some time today to figure out the VC-1/WMA part of the windows sample files, and pushed a rebased version here: http://gitorious.org/gstreamer/gst-plugins-good/commits/wip/qtdemux-fmp4 This is hopefully good enough for a first review. There are something like 14 patches. Some of them could use seperate bugs or could be applied directly, so the next rebase will be smaller. The most important patch is http://gitorious.org/gstreamer/gst-plugins-good/commit/c2cacc79722ac2667fbd6b07c7beef63290d134b I am not entirely satisfied with some of Andoni changes, because in theory, we should combine sample information from moov (the traditional header) and the various moof (the fragmented part). But at least I was happy to get it to play already, which means he didn't break it ;) I will be waiting for him to come back from holidays for improvements. Also, I don't know why progress bar doesn't work with totem.
thiagoss: I couldn't find any documentation about the vc1 and wma. So I made guesses based on what is exported in the smooth streaming Manifest (codec data + waveformatex). It might be incorrect, but at least gst-launch works with the *.ismv file provided by microsoft (I tested with several elements, gstreamer/ffmpeg and fluendo).
I've checked the patches and did some updates to them in the past few days. There are still some issues to work on here. e.g. push mode support and avoid using seeking to detect if the file is fragmented. But overall it's ok and it is good to have playback working for these files. So I'm pushing this after the releases. Hopefully we can get these issues sorted before the modules thaw and we have it all working by then.
<thiagoss> elmarco, btw, do you have any more links to samples of fragmented mp4? I'd like to do some further testing before pushing your patches :) <elmarco> thiagoss: yes, actually, they have changed small stuff <elmarco> thiagoss: for instance, they added a major brand isml <elmarco> although I can't find any documentation related to taht change <thiagoss> elmarco, could you put that info on the bug? I don't have time to discuss this now, so it is safer to have that on bugzilla. <elmarco> thiagoss: you can produce new files with EE4 (easy to find) <elmarco> thiagoss: I have a link with recent content as well, just a sec <elmarco> thiagoss: http://cid-bee3c9ac9541c85b.office.live.com/browse.aspx/.Public/Elephants^_Dream^_Mobile <elmarco> thiagoss: I didn't had time to verify the demux patch work with those new files <thiagoss> I'll do that this weel <thiagoss> *week. <thiagoss> I'll copy and paste this into the bug so I don't forget it :)
Created attachment 169912 [details] [review] qtdemux: make qtdemux_dump_mvhd parse version 1 correctly Versions 0 and 1 of mvhd have different sizes of its values (32bits/64bits). This patch makes it dump them correctly. Also use the right node in the parameter and not the root node.
Created attachment 169913 [details] [review] qtdemux: allow pulling atoms with unknown size Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 169914 [details] [review] qtdemux: parse the track id from the track header Signed-off-by: Andoni Morales Alastruey <amorales@flumotion.com>
Created attachment 169915 [details] [review] qtdemux: add fragmented mp4 fourccs Adds fourcc's for tfra, tfhd, trun, sdtp, trex, mehd and their dumps
Created attachment 169916 [details] [review] qtdemux: parse fmp4 samples information The fragmented mp4 format stores the tracks and samples information in the 'moof' boxes, which are appended before each fragment (fragment->'moof'+'mdat'). The 'mfra' box stores the offset of each 'moof' box and their presentation time. The location of this box can be retrieved from the 'mfro' box, which is located at the end of the file. The 'mfra' box is parsed to get the offset of each 'moof' box and their presentation time. Each 'moof' box can contain information for one or more tracks inside 'tfhd' boxes. For each track in a 'moof', we have a 'trun' box, which contains information of each sample (offset and duration) used to build the samples table. Based on patch by Marc-André Lureau <mlureau@flumotion.com>
Created attachment 169917 [details] [review] qtdemux: support wma & vc-1
Reviewed,reorganized and did some fixes to the patches. This now makes the BBB sample in the microsoft site play nicely when pull mode is available. Seeking is working too. These patches uses the pull function to get all the indexes spread on the multiple fragments and parse then before playing starts. As this is a streaming format, it would be really nice to have it working in push mode and parsing the indexes incrementally as they are found on the file.
Thanks for taking care of reviewing all the patches :) Having it working in push would be a big plus.
The muxer patches are now proposed in bug 632911.
So, based on the above work, I reorganized and fixed some more, the results [*] available in following repo/branch: http://git.collabora.co.uk/?p=user/manauw/gst-plugins-good.git;a=shortlog;h=refs/heads/fmp4 These should be almost "final", might do with a bit more testing (for which it would also help if I could find e.g. BBB MS sample content, which seems not so easy these days any more), but should be OK for both pull and push mode. The latter case, however, does not support seeking and reasonably expects the data in the file to be appropriate interleaved.
Thanks to an ismv sample coming my way, the above branch has been (non-fast-forward) updated to handle ismv quirks (along with some other fixes).
Branch now pushed to -good, so consists of quite some commits around/following: commit e7a1c32a4f3e850f154f98557563b9317142ef72 Author: Marc-André Lureau <mlureau@flumotion.com> Date: Thu Mar 11 15:34:49 2010 +0100 qtdemux: add fragmented mp4 fourccs Adds fourcc's for tfra, tfhd, trun, sdtp, trex, mehd and their dumps https://bugzilla.gnome.org/show_bug.cgi?id=596321
(In reply to comment #41) > Branch now pushed to -good, so consists of quite some commits around/following: Nice work. Thanks a lot!