GNOME Bugzilla – Bug 764629
qtdemux: Support sidx based seek in pull mode
Last modified: 2018-11-03 15:08:28 UTC
qtdemux: Support sidx based seek in pull mode tfra atom assists seeking to random access point. Because it is not mandatory, however, it is not desirable to rely only on it. In this context, sidx box can help seek for fragmented file. This patch enables qtdemux to seek for fragmented file if only sidx available (i.e., there is no tfra box in a file).
Created attachment 325412 [details] [review] qtdemux: Support sidx based seek in pull mode
Attaching logs - NG CASE 1. Seek around 1:00 min. 10414 0:00:03.193475147 23793 0x2570100 DEBUG GST_EVENT gstpad.c:5531:gst_pad_send_event_unchecked:<qtdemux0:video_0> have event type seek event: 0x7f336497a480, time 99:99:99.999999999, seq-num 795, GstEventSeek, rate=(double)1, format=(GstFormat)GST_FORMAT_TIME, flags=(GstSeekFlags)GST_SEEK_FLAG_FLUSH+GST_SEEK_FLAG_ACCURATE, cur-type=(GstSeekType)GST_SEEK_TYPE_SET, cur=(gint64)63462811074, stop-type=(GstSeekType)GST_SEEK_TYPE_SET, stop=(gint64)-1; 2. Activate segment 10507 0:00:03.229465206 23793 0x7f33681171e0 LOG qtdemux qtdemux.c:5538:gst_qtdemux_loop:<qtdemux0> loop at position 2643, state 2 10508 0:00:03.229493446 23793 0x7f33681171e0 INFO qtdemux qtdemux.c:5313:gst_qtdemux_loop_state_movie:<qtdemux0> pending fragmented seek 10509 0:00:03.229500011 23793 0x7f33681171e0 INFO qtdemux qtdemux.c:5315:gst_qtdemux_loop_state_movie:<qtdemux0> fragmented seek done! 10510 0:00:03.229505783 23793 0x7f33681171e0 LOG qtdemux qtdemux.c:1190:gst_qtdemux_find_segment:<qtdemux0:video_0> finding segment for 0:01:03.462811074 10511 0:00:03.229513452 23793 0x7f33681171e0 LOG qtdemux qtdemux.c:1198:gst_qtdemux_find_segment:<qtdemux0:video_0> looking at segment 0:00:00.000000000-0:02:33.194200000 10512 0:00:03.229521733 23793 0x7f33681171e0 LOG qtdemux qtdemux.c:4390:gst_qtdemux_activate_segment:<qtdemux0:video_0> activate segment 0, offset 0:01:03.462811074 10513 0:00:03.229528264 23793 0x7f33681171e0 LOG qtdemux qtdemux.c:4418:gst_qtdemux_activate_segment:<qtdemux0:video_0> seg_time 0:01:03.462811074 10514 0:00:03.229532969 23793 0x7f33681171e0 DEBUG qtdemux qtdemux.c:4462:gst_qtdemux_activate_segment:<qtdemux0:video_0> new segment 0 from 0:01:03.504522185 to 0:02:33.194200000, time 0:01:03.462811074 10515 0:00:03.229540302 23793 0x7f33681171e0 DEBUG qtdemux qtdemux.c:4483:gst_qtdemux_activate_segment:<qtdemux0:video_0> New segment: time segment start=0:01:03.504522185, offset=0:00:00.000000000, stop=0:02:33.194200000, rate=1.000000, applied_rate=1.000000, flags=0x01, time=0:01:03.462811074, base=0:00:00.000000000, position 0:01:03.504522185, duration 99:99:99.999999999 3. Because there is not moof offset info, push buffer from currently available moof 10951 0:00:03.231298749 23793 0x7f33681171e0 LOG qtdemux qtdemux.c:5160:gst_qtdemux_decorate_and_push_buffer:<qtdemux0> Pushing buffer with dts 0:00:25.025000000, pts 0:00:25.066711111, duration 0:00:00.041711111 on pad video_0 - With this patch 1. Seek around 1:00 min 9850 0:00:03.179459153 25197 0xcf10f0 DEBUG GST_EVENT gstpad.c:5531:gst_pad_send_event_unchecked:<qtdemux0:video_0> have event type seek event: 0x7fb664002660, time 99:99:99.999999999, seq-num 748, GstEventSeek, rate=(double)1, format=(GstFormat)GST_FORMAT_TIME, flags=(GstSeekFlags)GST_SEEK_FLAG_FLUSH+GST_SEEK_FLAG_ACCURATE, cur-type=(GstSeekType)GST_SEEK_TYPE_SET, cur=(gint64)63470510053, stop-type=(GstSeekType)GST_SEEK_TYPE_SET, stop=(gint64)-1; 2. qtdemux can figure out appropriate moof offset 9944 0:00:03.204426266 25197 0x7fb6ac0f9de0 LOG qtdemux qtdemux.c:5603:gst_qtdemux_loop:<qtdemux0> loop at position 2643, state 2 9945 0:00:03.204454235 25197 0x7fb6ac0f9de0 INFO qtdemux qtdemux.c:5378:gst_qtdemux_loop_state_movie:<qtdemux0> pending fragmented seek 9946 0:00:03.204460719 25197 0x7fb6ac0f9de0 LOG qtdemux qtdemux.c:5310:gst_qtdemux_do_fragmented_seek:<qtdemux0> Do fragmented seek using sidx 9947 0:00:03.204466196 25197 0x7fb6ac0f9de0 INFO qtdemux qtdemux.c:5327:gst_qtdemux_do_fragmented_seek:<qtdemux0:video_0> 0:01:00.060000000 at offset 101521121 9948 0:00:03.204474593 25197 0x7fb6ac0f9de0 INFO qtdemux qtdemux.c:5347:gst_qtdemux_do_fragmented_seek:<qtdemux0> seek to 0:01:03.470510053, best fragment moof offset: 101521121, ts 0:01:00.060000000 9949 0:00:03.204482545 25197 0x7fb6ac0f9de0 DEBUG qtdemux qtdemux.c:7599:qtdemux_add_fragmented_samples:<qtdemux0> next moof at offset 101521121 9950 0:00:03.204489026 25197 0x7fb6ac0f9de0 LOG qtdemux qtdemux.c:7539:qtdemux_find_atom:<qtdemux0> finding fourcc moof at offset 101521121 3. Push buffer from around 1:00 min 10399 0:00:03.206456971 25197 0x7fb6ac0f9de0 LOG qtdemux qtdemux.c:5208:gst_qtdemux_decorate_and_push_buffer:<qtdemux0> Pushing buffer with dts 0:01:00.060000000, pts 0:01:00.101711111, duration 0:00:00.041711111 on pad video_0
What's your use case for this, where do you get fragmented MP4 files with sidx outside the context of DASH?
(In reply to Sebastian Dröge (slomo) from comment #3) > What's your use case for this, where do you get fragmented MP4 files with > sidx outside the context of DASH? From the ISOBMFF specification, mp4 file support sidx box, and actually current qtdemux also use it for updating duration. The reported problem happen normal file play case. Due to regulation, I'm not sure the file can be shared or not....
Created attachment 325415 [details] problem file structure Attaching captured file structure
I mean where does it come from, what is it used for? :) It seems valid, yes, I'm just curious where this is used.
(In reply to Sebastian Dröge (slomo) from comment #6) > I mean where does it come from, what is it used for? :) It seems valid, yes, > I'm just curious where this is used. I'm not 100% sure where it came from.... I got this file from our QE testing server and found this file could not fully supported by current gstreamer (although this file seems follows ISOBMFF standard). This file is for testing media product
Maybe something like CFF/Ultraviolet? (Although I've never seen sidx in those, and if I remember correctly tfra is mandatory there, but it could be similar)
(In reply to Tim-Philipp Müller from comment #8) > Maybe something like CFF/Ultraviolet? (Although I've never seen sidx in > those, and if I remember correctly tfra is mandatory there, but it could be > similar) - Actually, tfra and sidx have very similar structure but sidx have some sophisticated additional info. - ISO/IEC 14496-12 document said tfra is not mandatory if my understanding is correct Definition Box Type: ‘tfra’ Container: Movie Fragment Random Access Box (‘mfra’) Mandatory: No Quantity: Zero or one per track Each entry contains the location and the presentation time of the random accessible sample. It indicates that the sample in the entry can be random accessed. Note that not every random accessible sample in the track needs to be listed in the table. The absence of this box does not mean that all the samples are sync samples. Random access information in the ‘trun’, ‘traf’ and ‘trex’ shall be set appropriately regardless of the presence of this box.
Review of attachment 325412 [details] [review]: Generally looks good, thanks. Just some remarks ::: gst/isomp4/qtdemux.c @@ +220,3 @@ GstClockTime ts; guint64 moof_offset; + gboolean ref_type; /* only used for sidx */ It might be nicer to directly use the GstSidxBox/GstSidxBoxEntries here. We might want to use more information from them later, and have them from the parser anyway @@ +2306,3 @@ + + if (stream->seg_idx_entries) + g_free (stream->seg_idx_entries); g_free() is NULL-safe, no need for the "if" here
Created attachment 326866 [details] [review] qtdemux: Support sidx based seek in pull mode
(In reply to Sebastian Dröge (slomo) from comment #10) > Review of attachment 325412 [details] [review] [review]: > > Generally looks good, thanks. Just some remarks > > ::: gst/isomp4/qtdemux.c > @@ +220,3 @@ > GstClockTime ts; > guint64 moof_offset; > + gboolean ref_type; /* only used for sidx */ > > It might be nicer to directly use the GstSidxBox/GstSidxBoxEntries here. We > might want to use more information from them later, and have them from the > parser anyway > ==> I tried to some ways to use "GstSidxBoxEntries" itself, but there is no benefit now by using it.... in my opinion. To do that, "GstSidxBoxEntries" and "QtDemuxRandomAccessEntry" might be decoupled. Rather than use "GstSidxBoxEntries", I unified parsed sidx box entry with "QtDemuxRandomAccessEntry" structure. Why don't we consider directly using of SidxBoxEntries when someone ask to use SidxBoxEntries for the other purpose? > @@ +2306,3 @@ > + > + if (stream->seg_idx_entries) > + g_free (stream->seg_idx_entries); > > g_free() is NULL-safe, no need for the "if" here ==> done.
Created attachment 370403 [details] [review] qtdemux: Support sidx based seek in pull mode
(In reply to Sebastian Dröge (slomo) from comment #10) > Review of attachment 325412 [details] [review] [review]: > It might be nicer to directly use the GstSidxBox/GstSidxBoxEntries here. We > might want to use more information from them later, and have them from the > parser anyway New patch uses GstSidxBox directly.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/264.