GNOME Bugzilla – Bug 780190
amcvideodec: Does not handle input without PTS correctly
Last modified: 2017-05-08 16:14:14 UTC
http://gstreamer-devel.966125.n4.nabble.com/HLS-stream-playback-is-not-smooth-with-1-11-2-vlc-ffplay-ok-td4682224.html Hi,I'm working with 1.11.2, when playing some HLS stream, obviously can feel the playback is not smooth, it should be drop frame. ---- .... 01-01 08:04:00.591 944 5883 W [gstbasesink.c:2901] <basesink> warning: A lot of buffers are being dropped. .... 01-01 08:04:02.042 944 5881 V [gstvideodecoder.c:3121] <videodecoder> accepting buffer inside segment: 0:00:02.043000000 0:00:02.084708333 seg 0:00:00.000000000 to 99:99:99.999999999 time 0:00:00.000000000 01-01 08:04:02.042 944 5881 D [gstvideodecoder.c:3155] <videodecoder> Dropping frame due to QoS. start:0:00:02.043000000 deadline:0:00:02.043000000 earliest_time:0:00:05.011717403 ... ---- On Android, just drop frames, the screen gives a slight sense of jitter. On PC, more serious, after about 45s,video frame freezes/hangs... Here is one of the HLS stream: https://drive.google.com/drive/folders/0B8t5E5lSOxhHbU1ERndPN3lFLU0?usp=sharing Drop frames should not be performance-related, video specifications are as follows: ------ Duration: 00:03:22.00, start: 567.273000, bitrate: 0 kb/s Program 0 Metadata: variant_bitrate : 0 Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 1200x504, 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc Metadata: variant_bitrate : 0 Stream #0:1: Audio: aac (HE-AAC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp Metadata: variant_bitrate : 0 ------ I extracted the source file audio/video pts with ffmpeg, the interval between the two frames are about 4000,it looks like normal. ffprobe -show_frames -select_streams a:0 -i http://10.9.44.131/issures/iqiyi/index.m3u8 > ~/Desktop/audio.info ffprobe -show_frames -select_streams v:0 -i http://10.9.44.131/issures/iqiyi/index.m3u8 > ~/Desktop/video.info ------ pts diff(PTS(n) - PTS(n-1)) pkt_pts 51057810 4230 pkt_pts 51062040 4140 pkt_pts 51066180 4230 pkt_pts 51070410 4140 .... pkt_pts 51166530 4140 pkt_pts 51170670 4230 pkt_pts 51174900 4140 .... ------- ，But gstreamer tsdemux.c gst_ts_demux_record_pts out of the pts is like this: ----- 01-02 01:18:21.227 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51153930 at offset 152092 01-02 01:18:21.403 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51229170 at offset 217892 <= diff= 51229170-51153930 = 75240 01-02 01:18:21.608 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51287670 at offset 280308 <= diff= 51287670 - 51229170 = 58500 01-02 01:18:21.649 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51316920 at offset 294220 <= diff = 51316920 - 51287670 = 29250 01-02 01:18:21.652 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51321150 at offset 294972 <= 51321150 - 51316920 = 4230 01-02 01:18:21.674 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51333660 at offset 304184 01-02 01:18:21.678 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51337890 at offset 305124 01-02 01:18:21.681 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51342030 at offset 305876 01-02 01:18:21.804 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51404760 at offset 352876 01-02 01:18:21.926 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51463260 at offset 392544 01-02 01:18:21.933 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51467400 at offset 393296 01-02 01:18:21.957 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51471630 at offset 400816 01-02 01:18:21.960 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51475770 at offset 401568 01-02 01:18:21.963 4289 4863 tsdemux.c:1973] <tsdemux> pid 0x0101 raw pts:51480000 at offset 402320 ----- More, (1) In https://drive.google.com/drive/folders/0B8t5E5lSOxhHbU1ERndPN3lFLU0?usp=sharing , Some ts file two adjacent pcr interval of 8s, which should be regarded as a problem well? 1. PCR:15314130000(00:09:27.190) PTS:51054570 DTS:51047100 2. PCR:15488685000(00:09:33.655) PTS:51636420 DTS:51628950 <= diff 6s 3. PCR:15713892000(00:09:41.996) PTS:52387200 DTS:52379640 <= diff 8s ... (2) m3u8 file containing EXT-X-DISCONTINUITY tags In tsdemux, the calculation of the timestamp is associated with the pcr, if the file itself pcr is not continuous, or the interval is too large, will lead to gstreamer A/V synchronization issure? Is it possible, do not use pcr to recalculate timestamp, just rely on pts / dts in the file as a basis for handling synchronization? Vlc and ffplay can smooth play, where is a problem gstreamer?May it be timestamp related? Or gstreamer can not properly handle EXT-X-DISCONTINUITY?
It should be androidmedia's a bug. In-depth investigation found that I provided stream(https://drive.google.com/drive/folders/0B8t5E5lSOxhHbU1ERndPN3lFLU0?usp=sharing), some frames without PTS (And it's key frames), as shown below： gstvideodecoder.c gst_video_decoder_decode_frame() -------------------------------------------- 01-02 09:29:54.345 PTS 0:00:00.083000000, DTS 0:00:00.000000000, dist 0 01-02 09:29:54.394 PTS 0:00:00.167000000, DTS 0:00:00.041000000, dist 1 01-02 09:29:54.412 PTS 0:00:00.125000000, DTS 0:00:00.083000000, dist 2 01-02 09:29:54.420 PTS 0:00:00.208000000, DTS 0:00:00.125000000, dist 3 .... 01-02 09:30:02.164 PTS 0:00:02.419000000, DTS 0:00:02.335000000, dist 56 01-02 09:30:02.227 PTS 0:00:02.377000000, DTS 0:00:02.376708333, dist 57 01-02 09:30:02.252 PTS 99:99:99.999999999, DTS 0:00:02.418416666, dist 58<=noPTS 01-02 09:30:02.298 PTS 0:00:02.544000000, DTS 0:00:02.461000000, dist 59 01-02 09:30:02.312 PTS 0:00:02.711000000, DTS 0:00:02.502000000, dist 60 .... 01-02 09:30:12.929 PTS 0:00:12.137000000, DTS 0:00:12.053000000, dist 289 01-02 09:30:12.957 PTS 0:00:12.095000000, DTS 0:00:12.094708333, dist 290 01-02 09:30:13.051 PTS 99:99:99.999999999, DTS 0:00:12.136416666, dist 291 <=no PTS .... ---------------------------------------- then,let us to track frame #58, gstvideodecoder.c gst_video_decoder_prepare_finish_frame(): ---------------------------------- 01-02 09:29:58.415 0xab85baa0 (#0) sync:1 PTS:0:00:00.083000000 DTS:0:00:00.000000000 01-02 09:29:58.430 0xab3643f0 (#2) sync:0 PTS:0:00:00.125000000 DTS:0:00:00.083000000 01-02 09:29:58.435 0xab85b9f8 (#1) sync:0 PTS:0:00:00.167000000 DTS:0:00:00.041000000 ... 01-02 09:30:39.734 0xab512690 (#873) sync:0 PTS:0:00:36.620000000 DTS:0:00:36.41100000 01-02 09:30:39.901 0xab512738 (#876) sync:0 PTS:0:00:36.578000000 DTS:0:00:36.53600000 01-02 09:30:39.903 0xab26cb48 (#879) sync:0 PTS:0:00:36.661000000 DTS:0:00:36.66170833 01-02 09:30:40.085 0xab512000 (#878) sync:0 PTS:0:00:36.703000000 DTS:0:00:36.62000000 01-02 09:30:40.086 0xac4675e8 (#880) sync:0 PTS:0:00:36.745000000 DTS:0:00:36.70300000 01-02 09:30:40.090 0xab26cbf0 (#877) sync:0 PTS:0:00:36.787000000 DTS:0:00:36.57800000 01-02 09:30:40.109 0xab3643f0 (#881) sync:0 PTS:0:00:36.953000000 DTS:0:00:36.74500000 01-02 09:30:40.111 0xab85baa0 (#58) sync:1 PTS:99:99:99.999999999 DTS:0:00:02.41841666 <= Ooops .... #58 traversed to about #881 .... This will cause GstVideoDecoder try to fix the timestamp, getting worse... 01-02 09:30:40.233 0xab364540 (#883) sync:0 PTS:0:00:36.828000000 DTS:0:00:36.82870833 01-02 09:30:40.235 0xab26cd40 (#882) sync:0 PTS:0:00:36.870000000 DTS:0:00:36.78700000 01-02 09:30:40.273 0xab26c9f8 (#884) sync:0 PTS:0:00:36.912000000 DTS:0:00:36.87000000 01-02 09:30:40.274 0xab3885e8 (#886) sync:0 PTS:0:00:37.037000000 DTS:0:00:36.95300000 01-02 09:30:40.366 0xac4673f0 (#887) sync:0 PTS:0:00:36.995000000 DTS:0:00:36.99470833 ---------------------------------- The reason for this issures is that the following code is logically wrong: /sys/androidmedia/gstamcvideodec.c @@ -682,7 +682,7 @@ _find_nearest_frame (GstAmcVideoDec * self, GstClockTime reference_timestamp) best_id = id; ... timestamp = id->timestamp; ... /* For frames without timestamp we simply take the first frame */ - if ((reference_timestamp == 0 && timestamp == 0) || diff == 0) As explained in the comment, if there is no PTS, we should take the first frame, and then let it have the opportunity to amend PTS in _prepare_finish_frame(). But,<timestamp == 0>should never be true, it should be compared with -1 rather than 0.
Created attachment 350442 [details] [review] amcvideodec-fix-no-PTS-_find_nearest_frame-logic-is-not-correct-issure Please review it.
Comment on attachment 350442 [details] [review] amcvideodec-fix-no-PTS-_find_nearest_frame-logic-is-not-correct-issure Looks correct but please configure your name correctly in GIT and then recreate the patch. The same change is also needed in gstamcvideoenc.c
Created attachment 350443 [details] [review] amcvideodec:fix no PTS _find_nearest_frame logic is not correct issure (In reply to Sebastian Dröge (slomo) from comment #3) > Comment on attachment 350442 [details] [review] [review] > amcvideodec-fix-no-PTS-_find_nearest_frame-logic-is-not-correct-issure > > Looks correct but please configure your name correctly in GIT and then > recreate the patch. > > The same change is also needed in gstamcvideoenc.c Thanks for your review. again.
commit febbaaddcf8de3eb1b5b27b17aea6cebec82a4b2 Author: shakin chou <shakin@outlook.com> Date: Wed Apr 26 17:46:10 2017 +0800 amcvideodec/enc: Correctly check for no PTS on input buffers MediaCodec gives us a presentation timestamp of 0 if it does not know anything, but GStreamer gives us GST_CLOCK_TIME_NONE. Don't mix up these two. https://bugzilla.gnome.org/show_bug.cgi?id=780190