GNOME Bugzilla – Bug 797257
baseparse/h264: basesink clipping one of the first NALUs because of DTS out of segment
Last modified: 2018-11-03 12:10:58 UTC
Created attachment 373871 [details] The log for the first buffers gst-launch-1.0 -v videotestsrc ! x264enc ! h264parse ! capsfilter caps="video/x-h264, stream-format=byte-stream, alignment=nal" ! fakesink silent=0 I enabled the logs for the parser and sink with GST_DEBUG=baseparse:7,h264parse:7,codecparsers_h264:7,basesink:7 to have the full picture. You can find the log for the first buffers attached. The sink is dropping the second NALU of the stream because it fells out of the segment. That's because, in absence of a valid pts, the basesink is clipping on the dts which i don't think is a good idea since dts can fall out of the segment (especially for the first frames of h264 streams with b-frames). The problem here comes also from the parser when transforming from a byte-stream format AU aligned to NAL aligned: when an AU is containing multiple NALU, only the first output buffer (NALU) is having the pts/dts of the input buffer(AU) while the following NALUs are having pts set to none and dts incremented by duration for every buffer. The attached log shows : - First input buffer to h264parse with size 7144: --> baseparse gstbaseparse.c:2189:gst_base_parse_handle_buffer:<h264parse0> handling buffer of size 7144 with dts 999:59:59.933333334, pts 1000:00:00.000000000, duration 99:99:99.999999999 - producing following buffers (NALUs): --> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 6 with dts 999:59:59.933333334, pts 1000:00:00.000000000, duration 0:00:00.033333333 --> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 33 with dts 999:59:59.966666667, pts 99:99:99.999999999, duration 0:00:00.033333333 --> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 9 with dts 1000:00:00.000000000, pts 99:99:99.999999999, duration 0:00:00.033333333 --> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 761 with dts 1000:00:00.033333333, pts 99:99:99.999999999, duration 0:00:00.033333333 --> baseparse gstbaseparse.c:2433:gst_base_parse_push_frame:<h264parse0> processing buffer of size 6335 with dts 1000:00:00.066666666, pts 99:99:99.999999999, duration 0:00:00.033333333 - Next input buffer to h264 with sie 5348: --> baseparse gstbaseparse.c:2189:gst_base_parse_handle_buffer:<h264parse0> handling buffer of size 5348 with dts 999:59:59.966666667, pts 1000:00:00.133333333, duration 99:99:99.999999999 The main problem is that the second NALU (with buffer size 6) is clipped out in the basesink because it have no valid pts and its dts is falling out of the segment (start: 1000:00:00.000000000). I'm not sure if it's better to set all NALUs to the same pts of the "parent" AU or set it to none, but the DTS is certainly set wrong. So the questions are : - Should the basesink ever clip based on the DTS (even if no PTS is valid) ? - What should be the (PTS,DTS) of the NALUs coming from the same AU ?
drop-out-of-segment=false from bug #765734 might be useful here fwiw.
Thanks for the hint. But I only used this pipeline to demonstrate the problem (as the behavior is related to basesink). In fact, my use case is a pipeline with an appsink that passes a NAL aligned buffers to a network packetizer which will reveal the NALU drop ( beside the "errounous" DTS ). Porting the fakesink patch to appsink ( or generalize it through basesink) can be a workaround for this kind of bugs but is clipping on DTS a wanted behavior for basesink ? And what kind of timestamps ( PTS, DTS) we want out of h264parse for NAL aligned buffers ?
This week I should be landing couple of parser improvement, one of the patch sets the proper dts on "header" buffers in NAL alignment. Note that the dts may still be out of segment, that I don't know. Is the segment expressed in DTS or PTS, because of the possible shift, one of the tow may fall out.
(header buffers are AUD, pps, sps, sei)
AFAIK, the segment is related to PTS ( that's why clipping to DTS does not seem a good behavior). Fixing the dts, for these Nalu types is definitely a good improvement but if we keep them to PTS (none) we will have the same issue as the basesink will assume ( in absence of valid PTS) that the dts should be used to segment clipping.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/issues/493.