GNOME Bugzilla – Bug 781069
postproc: framerate should not be always duplicated
Last modified: 2018-11-03 15:49:56 UTC
Created attachment 349532 [details] Gstreamer-VAAPI vs Gstreamer-MSDK When transcoding with Gstreamer-VAAPI a lot more of GPU resources are being used then Gstreamer-MSDK. Live Transcode VAAPI: GPU 27% Live Transcode MSDK: GPU 6% VAAPI seems to be using 4 times more GPU then MSDK. Please see the attached image I took screenshot of both servers. Environment: 2x identical servers with Xeon(R) CPU E3-1245 v5 SKYLAKE Server 1 with VAAPI: Ubuntu 16.04 vainfo: Driver version: Intel i965 driver for Intel(R) Skylake - 1.8.2.pre1 (1.7.3-372-g2f0a844) Gstreamer-VAAPI Pipeline: gst-launch-1.0 souphttpsrc location="http://localhost:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \ h264parse ! vaapih264dec ! vaapipostproc width=1280 height=720 ! vaapih264enc rate-control=2 bitrate=1700 ! h264parse ! \ flvmux streamable=true name=mux ! rtmpsink location="rtmp://localhost:1935/pushrtmp/vappi_outs live=1" demux. ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! \ mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux. Server 2 with MSDK: Centos 7.2 Media Server Studio 2017 vainfo: VA-API version: 0.99 (libva 1.67.0.pre1) Gstreamer-MSDK Pipeline: gst-launch-1.0 souphttpsrc location="http://localhost:80/oranews_HD/mpegts" is-live=true ! tsdemux name=demux ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 \ ! h264parse ! mfxh264dec ! mfxvpp width=1280 height=720 ! mfxh264enc rate-control=1 bitrate=1700 ! flvmux streamable=true name=mux ! rtmpsink location="rtmp://localhost:1935/pushrtmp/mfx_out live=1" demux. \ ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! mpegaudioparse ! queue ! avdec_mp2float plc=true ! audioconvert ! queue ! voaacenc bitrate=128000 ! mux.
Interesting, but I'm not sure there's much GStreamer can do about 2 internal driver implementations of video codec processing.
Thanks for reporting this issue. As Jan said, I don't know what we can do in GStreamer since this is related with the driver itself. I would recommend to start a discussion in the intel-vaapi-driver mailing list: https://lists.01.org/mailman/listinfo/intel-vaapi-media I'm closing it, but if you see that we can improve something in the gstreamer side, please reopen it.
Created attachment 349801 [details] VAAPI verbose pipeline
Hi, I have reported the issue with Intel-Vaapi-Driver they found out it's a GStreamer issue and told me to report back here. The findings were: Gstreamer is making the interlaced content double the framereate after inserting the deinterlacer. My source is intelaced at 25FPS the output comes out 50 FPS and that's what is making the load double in GPU. We tried disabling the deinterlace in vappipostproc and then load was normal and FPS was the same as source.
(In reply to Benjamin from comment #4) > Hi, > I have reported the issue with Intel-Vaapi-Driver they found out it's a > GStreamer issue and told me to report back here. > > The findings were: > Gstreamer is making the interlaced content double the framereate after > inserting the deinterlacer. My source is intelaced at 25FPS the output comes > out 50 FPS and that's what is making the load double in GPU. > > We tried disabling the deinterlace in vappipostproc and then load was normal > and FPS was the same as source. As far as I know, that's what interlaced media does[1], captures in a single frame, two different times. So, when doing the opposite, the framerate is duplicated. Am I losing something? 1. https://en.wikipedia.org/wiki/Interlaced
Frames are split, but it doesn't mean that if you deinterlace it the output to should be 50 FPS. I also tried to use deinterlce element and x264enc the output is the same as source stays at 25 FPS. same with mfxvpp and mfx264enc output stays at 25 FPS. but when using vaapipostproc and vaapih264enc output goes to 50 fps.
WE do the bob deinterlacing by default and the frame rate should be identical to field rate.
even Gstreamer-MSDK uses BOB but it's not outputing 50 FPS like VAAPI does. ! mfxh264dec ! mfxvpp width=1280 height=720 deinterlace-mode=1 ! mfxh264enc rate-control=1 bitrate=1700 ! when using MSDK output is normal FPS remains at 25 FPS with Deinterlacing. Gstreamer-VAAPI ! vaapih264dec ! vaapipostproc width=1280 height=720 deinterlace-method=1 ! vaapih264enc rate-control=2 bitrate=1700 ! when using VAAPI output goes to 50 FPS so the issue remains in Gstreamer-VAAPI
(In reply to sreerenj from comment #7) > WE do the bob deinterlacing by default and the frame rate should be > identical to field rate. I mean *It shouldn't double the framerate* :) If field rate is 30 , then framerate should also be 30. Because bob deinterlacing only do the line doubling(in each field picture) and we should get same number of frames
(In reply to sreerenj from comment #9) > (In reply to sreerenj from comment #7) > > WE do the bob deinterlacing by default and the frame rate should be > > identical to field rate. > > I mean *It shouldn't double the framerate* :) > > If field rate is 30 , then framerate should also be 30. Because bob > deinterlacing only do the line doubling(in each field picture) and we should > get same number of frames Hi Sree! Can you address me any documentation explaining that? Because all what a I've found [1][2] mention that (if I am reading correctly) bob, among other deinterlacing methods, is enabled, the frame rate is twice. Where the field rate is determined? 1. http://www.100fps.com/ 2. https://wiki.videolan.org/Deinterlacing
@Victor, to explain it in very simple way. In Interlacing the fields are split in single frame for speed in traditional broadcasting, it was done specially for sporting events to look smooth. Now when Deinterlacing the fields are doubled in the same frame and not doubling the same frame. here is video in reference to that: https://www.youtube.com/watch?v=YczLRshnxQ8&t=189s
(In reply to sreerenj from comment #9) > (In reply to sreerenj from comment #7) > > WE do the bob deinterlacing by default and the frame rate should be > > identical to field rate. > > I mean *It shouldn't double the framerate* :) > > If field rate is 30 , then framerate should also be 30. Because bob > deinterlacing only do the line doubling(in each field picture) and we should > get same number of frames Hm Wrong! I was thinking that it is "field encoded (not the two fields per frame scenario)" and framerate considered as field rate. But I don't think it is the case. IF the frame is encoded as two fields per frame(eg: 50 fields per second), and if the pipeline before deinterlacer report the frame rate as 25, then the output of deinterlacer should be 50fps. For eg: Try an interlaced content decode + deinterlace: vaapidecode ! vaapipostproc deintelace-method=1 avdec_h264 ! deinterlace method=6 ! Both should give same *doubled fps* after deinterlacer element.
I have used this as source for all my tests http://albcast.tv/vaapi/source_raw.ts I noticed that frame doubling happens only with Gstreamer's deinterlace element and VAAPI and the deinterlace is bad shaky video. then I tried avdeinterlace element from gst-libav and Gstreamer-MSDK and the video was good and framerate stayed the same as source please see pipelines below with downloadable output file. gst-launch-1.0 filesrc location="source_raw.ts" ! tsdemux ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse ! avdec_h264 ! deinterlace method=6 ! x264enc pass=0 bitrate=1700 ! mpegtsmux ! filesink location="avdec264_deintelace_6.ts" -v Output 50 FPS (not deinterlacing properly shaky video) http://albcast.tv/vaapi/avdec264_deintelace_6.ts gst-launch-1.0 filesrc location="source_raw.ts" ! tsdemux ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse ! avdec_h264 ! avdeinterlace ! x264enc pass=0 bitrate=1700 ! mpegtsmux ! filesink location="avdech264_avdeintelace.ts" -v ! avdec_h264 ! avdeinterlace ! x264enc pass=0 bitrate=1700 ! Output 25 FPS (GOOD video results) http://albcast.tv/vaapi/avdech264_avdeintelace.ts gst-launch-1.0 filesrc location="source_raw.ts" ! tsdemux ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse ! vaapih264dec ! vaapipostproc deinterlace-method=1 ! vaapih264enc rate-control=2 bitrate=1700 ! mpegtsmux ! filesink location="vaapih264dec_deintelace_1.ts" Output 50 FPS (not deinterlacing properly shaky video) http://albcast.tv/vaapi/vaapih264dec_deintelace_1.ts gst-launch-1.0 filesrc location="source_raw.ts" ! tsdemux ! queue max-size-buffers=1200 max-size-buffers=0 max-size-time=0 ! h264parse ! mfxh264dec ! mfxvpp deinterlace-mode=1 ! mfxh264enc rate-control=1 bitrate=1700 ! mpegtsmux ! filesink location="mfxh264dec_deinterlace_1.ts" Output 25 FPS (GOOD Video Results) http://albcast.tv/vaapi/mfxh264dec_deinterlace_1.ts Also I tried FFmpeg with deinterlacing it didn't double the framerate and no shaky video.
@Benjamin, to be in the same page, as far as I know there are several methods to deinterlace, but we can group them in two: 1) those which generate two frames with both fields, and 2) those which extract one field, and ditch the other. In the first group of methods (such as bob) from one original frame, we get two, thus the framerate is duplicated. When using the second group (such as weave) the framerate remains as original because there is one output frame from one input frame. Hence, we ought to ask which method is used in each element. In the particular case of vaapipostproc, by default, bob is used, which duplicates the framerate, just as the deinterlace element configured with the same method. Now, I'm not saying that vaapipostproc is bug free. I feel that we are not handling well the case of the second group of methods (discarding one frame). By the way, if you still want two frames (double framerate), but without shaking, use a better deinterlacing method such as motion-adaptive, though it is more expensive in terms of GPU.
@Victor, I tried every deinterlace option in vaapipostproc deinterlace-method : Deinterlace method to use flags: readable, writable Enum "GstVaapiDeinterlaceMethod" Default: 1, "bob" (0): none - Disable deinterlacing (1): bob - Bob deinterlacing (2): weave - Weave deinterlacing (3): motion-adaptive - Motion adaptive (4): motion-compensated - Motion compensated and they all doubling the framerate and GPU usage goes 20% just for deinterlacing alone. In the other hand i tried with https://github.com/01org/gstreamer-media-SDK gst-inspect-1.0 mfxvpp: deinterlacing with MSDK also uses bob by default deinterlace-mode : Deinterlace mode to use flags: readable, writable Enum "GstMfxDeinterlaceMode" Default: 1, "bob" (1): bob - Bob deinterlacing (2): adi - Advanced deinterlacing (11): adi-noref - Advanced deinterlacing (12): adi-scd - Advanced deinterlacing (13): weave - Field weaving Frame rate is not doubled with bob and video is not shaking. Also with MSDK deinterlcing has very low GPU footprint like 2% max. As of right now with VAAPI I can't do more then 3x 1920x1080@25 leaner transcodes GPU goes to 100% and it's Skylake HD Graphics 530 vs MSDK 12x 1920x1080@25
(In reply to Benjamin from comment #15) > @Victor, I tried every deinterlace option in vaapipostproc > > deinterlace-method : Deinterlace method to use flags: readable, > writable > Enum "GstVaapiDeinterlaceMethod" Default: 1, "bob" > (0): none - Disable deinterlacing > (1): bob - Bob deinterlacing > (2): weave - Weave deinterlacing > (3): motion-adaptive - Motion adaptive > (4): motion-compensated - Motion compensated > > and they all doubling the framerate and GPU usage goes 20% just for > deinterlacing alone. > > > In the other hand i tried with https://github.com/01org/gstreamer-media-SDK > gst-inspect-1.0 mfxvpp: > deinterlacing with MSDK also uses bob by default > > deinterlace-mode : Deinterlace mode to use > flags: readable, writable > Enum "GstMfxDeinterlaceMode" Default: 1, "bob" > (1): bob - Bob deinterlacing > (2): adi - Advanced deinterlacing > (11): adi-noref - Advanced deinterlacing > (12): adi-scd - Advanced deinterlacing > (13): weave - Field weaving > > Frame rate is not doubled with bob and video is not shaking. > Also with MSDK deinterlcing has very low GPU footprint like 2% max. > Can you check the performance with out the encoder in pipeline ? eg: a decdoer ! deinterlacer deinterlace-method=2 ! Just to check whether the driver is actually doing the deinterlacing and not doing the interlaced encoding over interlaced content. But most probably performance impact should be investigated from driver side. > > > As of right now with VAAPI I can't do more then 3x 1920x1080@25 leaner > transcodes GPU goes to 100% and it's Skylake HD Graphics 530 > > vs MSDK 12x 1920x1080@25 How many streams you can encode with gstreamer-vaapi, but with no de-interlacing?
deinterlacer without encoder: ! vaapih264dec ! vaapipostproc deinterlace-method=2 ! filesink location=capture1.raw Load: 24%, all this load just from deinterlacing. I asked the people in the driver side, they said the frame doubling is coming from Gstreamer-VAAPI not the driver and that's why there is high load. If I disable the deinterlacing I can do a lot more encoding.
I think there is a confusion in the interpretation of the field "framerate" here: We need a clarification on the interpretation of fps: If "fps" field in a decoder OR parser output caps supposed to represent: A)"actual output framerate in the display", then a fix needed in GStreamer-vaapi. Don't double the framerate. B)"output framerate(not field-rate) producing by the parser/decoder", then a fix needed in parser/decoder to make the fps as half of field-rate. C)"output field-rate producing by the parser/decoder", then a fix needed in GStreamer-vaapi, Don't double the framerate. But we need a documentation in GStreamer explaining "field-rate represents the framerate". Note: Upstream gst-MSDK won't duplicate the fps, because it requires the field rate as input framerate(which is what the parser is providing and inherited to decoder too) and actual frame rate as output framerate (which is equal to field-rate).
In gstreamer caps, the framerate field is always frames. So the parser helper gst_h264_video_calculate_framerate() returns a framerate (it will half the SPS framerate). Though, if I understood correctly, for scalerbob technique, if you don't double the rate, it means we will likely scale up all top fields, or all bottom fields, dropping the other fields. Is that what MSDK VPP implementation do ? This would obviously remove shakyness.
(In reply to Nicolas Dufresne (ndufresne) from comment #19) > In gstreamer caps, the framerate field is always frames. So the parser > helper gst_h264_video_calculate_framerate() returns a framerate (it will > half the SPS framerate). > So the parser produces fps which is the actual number of frames outputting from the parser, it won't send a field in a buffer I guess. Then we all are good, gstreamer-vaapi should double the framerate IMO. > Though, if I understood correctly, for scalerbob technique, if you don't > double the rate, it means we will likely scale up all top fields, or all > bottom fields, dropping the other fields. Is that what MSDK VPP > implementation do ? This would obviously remove shakyness. I'm not sure what msdk internally do, but it provides two modes for BOB. Either you can output half-framerate or double-framerate. It is up to the user to chose what he/she wants. What I am curious is, whether the driver always doubles the fps or not. Is it only msdk plays in the middle to drop or keep frames? We are dealing with two drivers here. msdk uses iHD and gst-vaapi uses i965. It is worth to investigate whether one driver has a mode to set the number of frames required for deinterlacing based on user-provided fps.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/issues/54.