GNOME Bugzilla – Bug 797280
webrtcbin: video pixelated
Last modified: 2018-11-03 14:36:08 UTC
Hello. I using gstwebrtc-demos for experiments. I no modifications this demo, video are chronically pixelated. My run demo on Intel Core i5-2300. Tested on Fedora 28 and AltLinux 7. GStreamer 1.14.1-1.14.4 and git master branch. On localhost for vp8: chrome -> gstreamer video pixeleted after few seconds and not restoring. FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain() firefox -> gstreamer not pixelated. Via internet for vp8: chrome -> gstreamer video pixeleted after few seconds and not restoring. FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain() firefox -> gstreamer pixelated periodical and restoring. FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain() On localhost for vp9: chrome -> gstreamer video not pixeleted, after one minute decode error. Logged vp9dec0 Failed to decode frame firefox -> gstreamer not pixelated. Via internet for vp9: chrome -> gstreamer video not pixeleted, after one minute decode error. Logged vp9dec0 Failed to decode frame firefox -> gstreamer pixelated periodical and restoring. FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain() On localhost for h264: chrome -> gstreamer not working. Via wi-fi for h264: ios safari -> gstreamer pixelated periodical and restoring. I modified demo for transfer video hd 720p. I took 720p video clip (~ 2000 kbit/s), split the video and audio into different files. Replace videotestsrc on filesrc ! decodebin. On localhost: chrome -> gstremer video pixeleted after few seconds and not restoring. gstreamer -> chrome video from file normally reproduced in good quality and not pixelated. Periodically, the video freezes(slows down), but continues to play. I modified demo for connection gstreamer <-> gstreamer and show packets-lost statistics. On localhost on CPU Intel Core i5-2300 both videos pixelated. On localhost on CPU Intel Core i7-7500U not pixelated, video can play for hours. But sometimes it is suddenly pixelated and not restored. Via internet video starts to pixelate even without packet loss. Without vaapi logged: FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain() For vaapi logged: 0:01:05.094888406 31361 0x164a190 WARN basesink gstbasesink.c:3008:gst_base_sink_is_too_late:<sink> warning: A lot of buffers are being dropped. 0:01:05.094911959 31361 0x164a190 WARN basesink gstbasesink.c:3008:gst_base_sink_is_too_late:<sink> warning: There may be a timestamping problem, or this computer is too slow. 0:01:09.812852771 31361 0x164a190 WARN basesink gstbasesink.c:3008:gst_base_sink_is_too_late:<sink> warning: A lot of buffers are being dropped. 0:01:09.812895157 31361 0x164a190 WARN basesink gstbasesink.c:3008:gst_base_sink_is_too_late:<sink> warning: There may be a timestamping problem, or this computer is too slow. or 0:00:46.437781309 30576 0x7fcdbc086a80 WARN rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:21.974949098 too big, reset skew 0:00:54.432961894 30576 0x7fcdbc086a80 WARN rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:07.929154280 too big, reset skew 0:00:55.091775442 30576 0x7fcdbc086a80 WARN rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:01.005567476 too big, reset skew 0:00:55.751770789 30576 0x7fcdbc086a80 WARN rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:01.006104407 too big, reset skew 0:00:56.032690630 30576 0x7fcd8c0044a0 WARN videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.420374279 < 0:00:47.392941755) 0:00:56.035278829 30576 0x7fcd8c0044a0 WARN videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.454371755 < 0:00:47.392941755) 0:00:56.037620125 30576 0x7fcd8c0044a0 WARN videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.487322322 < 0:00:47.392941755) 0:00:56.039805150 30576 0x7fcd8c0044a0 WARN videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.520118718 < 0:00:47.392941755) 0:00:56.042165373 30576 0x7fcd8c0044a0 WARN videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.553719923 < 0:00:47.392941755) I tried to minimize the load on the processor. I check gstreamer(h264) <-> gstreamer(h264) via internet using vaapi for decoder and encoder. Video is almost not pixelated. Only sometimes and immediately restored. Packet loss is about the same as in previous tests. How to make the video just slow down when there is a shortage of computer resources and then continued to play?
Do you observe high CPU usage with the unmodified version of the webrtc demo? Have you tried modifying the media constraints in webrtc.js to a lower resolution? Pixelated is not a very good description, do you mean chrome lowers the quality it sends or the decoded video is actually corrupted? If the video is corrupted, that means gstreamer hasn't received a keyframe and should be requesting a new one through RTCP. If the quality is lowered, it means chrome has probably detected packet loss (based on the receiver reports sent by gstreamer through RTCP), maybe the statistics are affected when the pipeline struggles to perform decoding in real time.
(In reply to Mathieu Duponchelle from comment #1) > Do you observe high CPU usage with the unmodified version of the webrtc demo? Intel Core i5-2300 not support vp8 and vp9. Without vaapi used vp8dec/vp8dec and CPU usage 150% - 250%. Pixelated always. Intel Core i7-7500U support vp8 and vp9. Without vaapi used vp8dec/vp8dec and CPU usage 150% - 250%. With vaapi ~10%. Pixelated sometimes. Intel Core i5-2300 my working computer, everything works quickly, it suits me for many years and to this day. Watch YouTube, Skype without problems, webrtc browser to browser no problem. > Have you tried modifying the media constraints in webrtc.js to a lower resolution? { "video": true, "audio": true } - no problem { "video": { "width": { "exact": 1280 }, "height": { "exact": 800 } }, "audio": true } - pixelated > Pixelated is not a very good description, do you mean chrome lowers the quality it sends or the decoded video is actually corrupted? The image is degradated and is not restored. I can not see anything. > If the video is corrupted, that means gstreamer hasn't received a keyframe and should be requesting a new one through RTCP. According to my observations, this is most likely what happens. In the case of packet loss, the image is pixelated and immediately restored. Often not pixelated, I just see in the statistics a packet loss, but the image is good on CPU Intel Core i7-7500U with vaapi. > If the quality is lowered, it means chrome has probably detected packet loss (based on the receiver reports sent by gstreamer through RTCP), maybe the statistics are affected when the pipeline struggles to perform decoding in real time. In chrome video good, no problem. Video pixelated in gstreamer from camera on chrome. Pixelated when gstreamer <-> gstreamer. I think the problem is between webrtcbin ! decodebin . I also described this problem in issee on github https://github.com/centricular/gstwebrtc-demos/issues/31
Created attachment 373914 [details] Pixelated video showed in autovideosink
We'll need to create a new word for that. Maroblockelated ? The drain() warning happens on gaps and discont(), so clearly there is some lost. Something to keep in mind is that GStreamer does not implement the Google rate control feedback. That usually mean a highly conservative rate and deadline is pick, combined with possible storm of keyframe request (can be check by monitoring the RTCP flow), it can easily fall apart.
I think you should share the encoding parameters that you are using, vaapi default are aiming constant quality, so they generally behaves better then vpxenc default.
(In reply to Nicolas Dufresne (ndufresne) from comment #4 and #5) > We'll need to create a new word for that. Maroblockelated ? Excuse me. I'm from Russia. I don't speak English. I do not understand the meaning of this word. Please explain. > The drain() warning happens on gaps and discont(), so clearly there is some lost. But why the video is not restored. I admit the short-term appearance of artifacts, but it should be restored immediately. > Something to keep in mind is that GStreamer does not implement the Google rate control feedback. That usually mean a highly conservative rate and deadline is pick, combined with possible storm of keyframe request (can be check by monitoring the RTCP flow), it can easily fall apart. I do not think that this is somehow related to this problem. After all, the problem has a place to be at communication gstreamer <-> gstreamer. In gstreamer exists FEC and NACK-PLI. > I think you should share the encoding parameters that you are using, vaapi default are aiming constant quality, so they generally behaves better then vpxenc default. The problem is not in encoding, but in decoding. This should not be a problem. gstreamer <-> gstreamer vp8enc deadline=1. for acceleration i using vaapih264enc without parameters. Chrome encoding parameters I can't know. Problem in gstreamer decoding. It seems to me to understand the problem about which I speak you just need to reproduce it. To reproduce it need: 1. Take any mediocre computer, such as the Intel Core i5-2300(I tried others). 2. Install the OS, for example Fedora 28(I tried AltLinux 7, irrelevant). 3. Compile gstwebrtc-demo. 4. Open in chrome https://webrtc.nirbheek.in 5. Specify the best quality for your camera constraints {"video": {"width": {"exact": 1280}, "height": {"exact": 800}}, "audio": true} 6. Run everything on one computer, localhost connection. I always have a problem. Is it not reproduced by anyone?
(In reply to Dmitriy from comment #6) > I always have a problem. Is it not reproduced by anyone? It might be reproduced by others who have not taken the time to file an issue, or commented here ;) I do not see this issue on my development machine, but it does not qualify as mediocre. What Nicolas suggested is the first thing you might want to look into in my opinion, does gstreamer request new keyframes? Are the requests transmitted to chrome? Do we actually receive new keyframes? If you want to see the relevant logs in chromium, you can use something like: chromium --enable-logging=stderr --vmodule=*/webrtc/*=6,*/libjingle/*=2,*=9 --no-sandbox , it is very verbose but might show something interesting.
I run gstwebrtc-demo on notebook Samsung RC530 Intel Core i5-2430M with constraints {"video": {"width": {"exact": 1280}}, "audio": true}. Video pixelated after ~20 second. $ chromium-browser --enable-logging=stderr --vmodule=*/webrtc/*=6,*/libjingle/*=2,*=9 --no-sandbox 2>&1 | tee ~/chromium.log $ GST_DEBUG=5 ./webrtc-sendrecv --peer-id=806 2>&1 | tee ~/gstreamer.log
Created attachment 373916 [details] chromium.log
Created attachment 373917 [details] gstreamer.log.xz.part-0
Created attachment 373918 [details] gstreamer.log.xz.part-1
Created attachment 373919 [details] gstreamer.log.xz.part-2
Created attachment 373920 [details] gstreamer.log.xz.part-3
Created attachment 373921 [details] gstreamer.log.xz.part-4
So, my notes. We can see that chrome raises the bitrate, and resolution, till it reaches 720p 1.2MB. The where you start having packet lost. Then it reduces the frame rate aggressively before reducing the encoding rate to about 1Mb, after that you have stopped everything. There is warning about not getting RTCP RR in time, would need investigation. I don't see any trace that would indicate that retransmission is enabled. Does that demo advertise this ? If you aren't using some sort of FEC or transmission, the rate adaptation will always lead to visual corruption. There is not trace that would indicate a keyframe request, if this is broken, obviously this is a problem. I'm not sure how to interpre the hitogram traces. Histogram: WebRTC.Video.KeyFramesSentInPermille recorded 1 samples, mean = 12,0 (flags = 0x41) Does this mean it has sent 12 keyframes ? Maybe keyframe at the edge rate just never make it. Btw, on slow computer, you have packet lost whenever the CPU is too busy to dequeue the packet from the UDP socket, it's not always related to the network capacity. I don't think that chrome algo is really designed to cope with that. Though, you can increase the queue size (see buffer-size on udpsrc), and then sysctl net.core.rmem_max=value to ask the kernel to allow more then the small default. You could also just try and add a queue after udpsrc, not sure how that work with webrtcbin (if it's not already the case). There is already a queue of course, it's the rtpjitterbuffer, but it does not perform very well iirc on loaded systems. (Haven't looked at the gst logs yet)
(In reply to Nicolas Dufresne (ndufresne) from comment #15) > There is warning about not getting RTCP RR in time, would need > investigation. I don't see any trace that would indicate that retransmission > is enabled. Does that demo advertise this ? If you aren't using some sort of > FEC or transmission, the rate adaptation will always lead to visual > corruption. Webrtcbin does not create in sdp fec, ulpfec, red in offer/answer. How to use fec with webrtcbin? > Btw, on slow computer, you have packet lost whenever the CPU is too busy to > dequeue the packet from the UDP socket, it's not always related to the > network capacity. I don't think that chrome algo is really designed to cope > with that. Though, you can increase the queue size (see buffer-size on > udpsrc), and then sysctl net.core.rmem_max=value to ask the kernel to allow > more then the small default. You could also just try and add a queue after > udpsrc, not sure how that work with webrtcbin (if it's not already the > case). There is already a queue of course, it's the rtpjitterbuffer, but it > does not perform very well iirc on loaded systems. Even if you adjust the parameters of the kernel, even though it will bear fruit, but this is not a solution. Packet loss is inevitable. Also, the gstreamer can be run on a machine on which there is no possibility to change the parameters of the kernel. If a lot of independent video streams are launched on one machine and the machine for some reason or other slows down, the image will broken in all streams at once. And it will not depend on the characteristics of the processor. No matter how powerful it may be, the problem will be!
I did the FEC as shown in the example https://github.com/GStreamer/gst-plugins-bad/blob/master/tests/examples/webrtc/webrtctransceiver.c sdp: a=rtpmap:96 VP8/90000 a=rtcp-fb:96 nack pli a=framerate:30 a=rtpmap:97 red/90000 a=rtpmap:98 ulpfec/90000 Problem still exists. The video also pixelated.
Created attachment 374112 [details] Chrome stats
I was on vacation, so it was not possible to experiment. I tried to configure the linux kernel, but that did not change anything. Made more convenient output of data on packet loss from get-stats. Video pixelated immediately after the first loss. Next, the packet lost counter increases by about 100 per second. This goes on for a few minutes. Then the video is restored and the packet lost counter stops increasing. After a few seconds, everything repeats. Changing net.core.rmem_max gives nothing. sysctl -w net.core.rmem_max=16777216 I compared the data from the get-stats with the data from /proc/net/udp. # cat /proc/net/udp | grep ':DDED' sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops 1060: 8201A8C0:DDED 00000000:0000 07 00000000:00000000 00:00000000 00000000 1000 0 1202334542 2 000000009ee2ba36 10603 From get-stats: Inbound RTP Stats: ssrc: 1183057958 fir-count: 0 pli-count: 0 nack-count: 0 packets-received: 791085 bytes-received: 917999656 jitter: 1 packets-lost: 98057 /proc/net/udp in 10 times less. Packet lost counter increases by about 100 per second, but value drops in /proc/net/udp changes extremely rarely, no correlation. All of the above said about CPU Intel Core i5-2300 and chrome <-> gstreamer without vaapi. On CPU Intel Core i7-7500U in /proc/net/udp always 0, but after ~10 minute video pixelated, get-stats show packets-lost ~ 100 in sec and /proc/net/udp everything is 0. Why ~20 sec no packet drops and then packet lost regularly? The chrome graphics show that it does not send huge frames. Nack PLI not working. Why decodebin not send event GstForceKeyUnit? How to make it work?
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/799.