After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 797280 - webrtcbin: video pixelated
webrtcbin: video pixelated
Status: RESOLVED OBSOLETE
Product: GStreamer
Classification: Platform
Component: gst-plugins-bad
unspecified
Other Linux
: Normal normal
: git master
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2018-10-12 11:12 UTC by Dmitriy
Modified: 2018-11-03 14:36 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Pixelated video showed in autovideosink (1.30 MB, image/png)
2018-10-13 06:50 UTC, Dmitriy
Details
chromium.log (1.96 MB, text/x-log)
2018-10-13 19:10 UTC, Dmitriy
Details
gstreamer.log.xz.part-0 (3.00 MB, application/octet-stream)
2018-10-13 19:20 UTC, Dmitriy
Details
gstreamer.log.xz.part-1 (3.00 MB, application/octet-stream)
2018-10-13 19:21 UTC, Dmitriy
Details
gstreamer.log.xz.part-2 (3.00 MB, application/octet-stream)
2018-10-13 19:22 UTC, Dmitriy
Details
gstreamer.log.xz.part-3 (3.00 MB, application/octet-stream)
2018-10-13 19:23 UTC, Dmitriy
Details
gstreamer.log.xz.part-4 (423.89 KB, application/octet-stream)
2018-10-13 19:24 UTC, Dmitriy
Details
Chrome stats (348.54 KB, image/jpeg)
2018-10-31 04:16 UTC, Dmitriy
Details

Description Dmitriy 2018-10-12 11:12:32 UTC
Hello.

I using gstwebrtc-demos for experiments.
I no modifications this demo, video are chronically pixelated.

My run demo on Intel Core i5-2300. Tested on Fedora 28 and AltLinux 7.
GStreamer 1.14.1-1.14.4 and git master branch.

On localhost for vp8:
chrome -> gstreamer video pixeleted after few seconds and not restoring. FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain()
firefox -> gstreamer not pixelated.

Via internet for vp8: 
chrome -> gstreamer video pixeleted after few seconds and not restoring. FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain()
firefox -> gstreamer pixelated periodical and restoring. FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain()

On localhost for vp9:
chrome -> gstreamer video not pixeleted, after one minute decode error. Logged vp9dec0 Failed to decode frame
firefox -> gstreamer not pixelated.

Via internet for vp9: 
chrome -> gstreamer video not pixeleted, after one minute decode error. Logged vp9dec0 Failed to decode frame
firefox -> gstreamer pixelated periodical and restoring.  FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain()

On localhost for h264:
chrome -> gstreamer not working.

Via wi-fi for h264: 
ios safari -> gstreamer pixelated periodical and restoring.

I modified demo for transfer video hd 720p. I took 720p video clip (~ 2000 kbit/s), split the video and audio into different files.
Replace videotestsrc on filesrc ! decodebin.

On localhost:
chrome -> gstremer  video pixeleted after few seconds and not restoring.
gstreamer -> chrome video from file normally reproduced in good quality and not pixelated. Periodically, the video freezes(slows down), but continues to play.

I modified demo for connection gstreamer <-> gstreamer and show packets-lost statistics.

On localhost on CPU Intel Core i5-2300  both videos pixelated.
On localhost on CPU Intel Core i7-7500U not pixelated, video can play for hours. But sometimes it is suddenly pixelated and not restored.

Via internet video starts to pixelate even without packet loss.
Without vaapi logged:
FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out: Sub-class should implement drain()

For vaapi logged:
0:01:05.094888406 31361      0x164a190 WARN                basesink gstbasesink.c:3008:gst_base_sink_is_too_late:<sink> warning: A lot of buffers are being dropped.
0:01:05.094911959 31361      0x164a190 WARN                basesink gstbasesink.c:3008:gst_base_sink_is_too_late:<sink> warning: There may be a timestamping problem, or this computer is too slow.
0:01:09.812852771 31361      0x164a190 WARN                basesink gstbasesink.c:3008:gst_base_sink_is_too_late:<sink> warning: A lot of buffers are being dropped.
0:01:09.812895157 31361      0x164a190 WARN                basesink gstbasesink.c:3008:gst_base_sink_is_too_late:<sink> warning: There may be a timestamping problem, or this computer is too slow.

or

0:00:46.437781309 30576 0x7fcdbc086a80 WARN         rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:21.974949098 too big, reset skew
0:00:54.432961894 30576 0x7fcdbc086a80 WARN         rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:07.929154280 too big, reset skew
0:00:55.091775442 30576 0x7fcdbc086a80 WARN         rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:01.005567476 too big, reset skew
0:00:55.751770789 30576 0x7fcdbc086a80 WARN         rtpjitterbuffer rtpjitterbuffer.c:570:calculate_skew: delta - skew: 0:00:01.006104407 too big, reset skew
0:00:56.032690630 30576 0x7fcd8c0044a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.420374279 < 0:00:47.392941755)
0:00:56.035278829 30576 0x7fcd8c0044a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.454371755 < 0:00:47.392941755)
0:00:56.037620125 30576 0x7fcd8c0044a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.487322322 < 0:00:47.392941755)
0:00:56.039805150 30576 0x7fcd8c0044a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.520118718 < 0:00:47.392941755)
0:00:56.042165373 30576 0x7fcd8c0044a0 WARN            videodecoder gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame:<vp8dec0> decreasing timestamp (0:00:46.553719923 < 0:00:47.392941755)


I tried to minimize the load on the processor. I check gstreamer(h264) <-> gstreamer(h264) via internet using vaapi for decoder and encoder.
Video is almost not pixelated. Only sometimes and immediately restored. Packet loss is about the same as in previous tests.


How to make the video just slow down when there is a shortage of computer resources and then continued to play?
Comment 1 Mathieu Duponchelle 2018-10-12 21:24:21 UTC
Do you observe high CPU usage with the unmodified version of the webrtc demo?

Have you tried modifying the media constraints in webrtc.js to a lower resolution?

Pixelated is not a very good description, do you mean chrome lowers the quality it sends or the decoded video is actually corrupted?

If the video is corrupted, that means gstreamer hasn't received a keyframe and should be requesting a new one through RTCP.

If the quality is lowered, it means chrome has probably detected packet loss (based on the receiver reports sent by gstreamer through RTCP), maybe the statistics are affected when the pipeline struggles to perform decoding in real time.
Comment 2 Dmitriy 2018-10-13 06:48:49 UTC
(In reply to Mathieu Duponchelle from comment #1)
> Do you observe high CPU usage with the unmodified version of the webrtc demo?
Intel Core i5-2300 not support vp8 and vp9. Without vaapi used vp8dec/vp8dec and CPU usage 150% - 250%. Pixelated always.
Intel Core i7-7500U support vp8 and vp9. Without vaapi used vp8dec/vp8dec and CPU usage 150% - 250%. With vaapi ~10%. Pixelated sometimes.

Intel Core i5-2300 my working computer, everything works quickly, it suits me for many years and to this day. Watch YouTube, Skype without problems, webrtc browser to browser no problem.

> Have you tried modifying the media constraints in webrtc.js to a lower resolution?
{ "video": true, "audio": true } - no problem
{ "video": { "width": { "exact": 1280 }, "height": { "exact": 800 } }, "audio": true } - pixelated

> Pixelated is not a very good description, do you mean chrome lowers the quality it sends or the decoded video is actually corrupted?
The image is degradated and is not restored. I can not see anything.

> If the video is corrupted, that means gstreamer hasn't received a keyframe and should be requesting a new one through RTCP.
According to my observations, this is most likely what happens. In the case of packet loss, the image is pixelated and immediately restored. Often not pixelated, I just see in the statistics a packet loss, but the image is good on CPU Intel Core i7-7500U with vaapi.


> If the quality is lowered, it means chrome has probably detected packet loss (based on the receiver reports sent by gstreamer through RTCP), maybe the statistics are affected when the pipeline struggles to perform decoding in real time.
In chrome video good, no problem. Video pixelated in gstreamer from camera on chrome. Pixelated when gstreamer <-> gstreamer. I think the problem is between webrtcbin ! decodebin .

I also described this problem in issee on github https://github.com/centricular/gstwebrtc-demos/issues/31
Comment 3 Dmitriy 2018-10-13 06:50:34 UTC
Created attachment 373914 [details]
Pixelated video showed in autovideosink
Comment 4 Nicolas Dufresne (ndufresne) 2018-10-13 13:25:12 UTC
We'll need to create a new word for that. Maroblockelated ?

The drain() warning happens on gaps and discont(), so clearly there is some lost. 

Something to keep in mind is that GStreamer does not implement the Google rate control feedback. That usually mean a highly conservative rate and deadline is pick, combined with possible storm of keyframe request (can be check by monitoring the RTCP flow), it can easily fall apart.
Comment 5 Nicolas Dufresne (ndufresne) 2018-10-13 13:27:31 UTC
I think you should share the encoding parameters that you are using, vaapi default are aiming constant quality, so they generally behaves better then vpxenc default.
Comment 6 Dmitriy 2018-10-13 14:58:31 UTC
(In reply to Nicolas Dufresne (ndufresne) from comment #4 and #5)
> We'll need to create a new word for that. Maroblockelated ?
Excuse me. I'm from Russia. I don't speak English. I do not understand the meaning of this word. Please explain.

> The drain() warning happens on gaps and discont(), so clearly there is some lost.
But why the video is not restored. I admit the short-term appearance of artifacts, but it should be restored immediately.

> Something to keep in mind is that GStreamer does not implement the Google rate control feedback. That usually mean a highly conservative rate and deadline is pick, combined with possible storm of keyframe request (can be check by monitoring the RTCP flow), it can easily fall apart.
I do not think that this is somehow related to this problem. After all, the problem has a place to be at communication gstreamer <-> gstreamer. In gstreamer exists FEC and NACK-PLI.

> I think you should share the encoding parameters that you are using, vaapi default are aiming constant quality, so they generally behaves better then vpxenc default.
The problem is not in encoding, but in decoding. This should not be a problem.
gstreamer <-> gstreamer vp8enc deadline=1. for acceleration i using vaapih264enc without parameters.
Chrome encoding parameters I can't know. Problem in gstreamer decoding.

It seems to me to understand the problem about which I speak you just need to reproduce it.

To reproduce it need:
1. Take any mediocre computer, such as the Intel Core i5-2300(I tried others).
2. Install the OS, for example Fedora 28(I tried AltLinux 7, irrelevant).
3. Compile gstwebrtc-demo.
4. Open in chrome https://webrtc.nirbheek.in
5. Specify the best quality for your camera constraints {"video": {"width": {"exact": 1280}, "height": {"exact": 800}}, "audio": true}
6. Run everything on one computer, localhost connection.

I always have a problem. Is it not reproduced by anyone?
Comment 7 Mathieu Duponchelle 2018-10-13 17:55:11 UTC
(In reply to Dmitriy from comment #6)

> I always have a problem. Is it not reproduced by anyone?

It might be reproduced by others who have not taken the time to file an issue, or commented here ;) I do not see this issue on my development machine, but it does not qualify as mediocre.

What Nicolas suggested is the first thing you might want to look into in my opinion, does gstreamer request new keyframes? Are the requests transmitted to chrome? Do we actually receive new keyframes?

If you want to see the relevant logs in chromium, you can use something like: chromium --enable-logging=stderr --vmodule=*/webrtc/*=6,*/libjingle/*=2,*=9 --no-sandbox , it is very verbose but might show something interesting.
Comment 8 Dmitriy 2018-10-13 19:09:16 UTC
I run gstwebrtc-demo on notebook Samsung RC530 Intel Core i5-2430M with constraints {"video": {"width": {"exact": 1280}}, "audio": true}.

Video pixelated after ~20 second.

$ chromium-browser --enable-logging=stderr --vmodule=*/webrtc/*=6,*/libjingle/*=2,*=9 --no-sandbox 2>&1 | tee ~/chromium.log
$ GST_DEBUG=5 ./webrtc-sendrecv --peer-id=806 2>&1 | tee ~/gstreamer.log
Comment 9 Dmitriy 2018-10-13 19:10:26 UTC
Created attachment 373916 [details]
chromium.log
Comment 10 Dmitriy 2018-10-13 19:20:06 UTC
Created attachment 373917 [details]
gstreamer.log.xz.part-0
Comment 11 Dmitriy 2018-10-13 19:21:17 UTC
Created attachment 373918 [details]
gstreamer.log.xz.part-1
Comment 12 Dmitriy 2018-10-13 19:22:15 UTC
Created attachment 373919 [details]
gstreamer.log.xz.part-2
Comment 13 Dmitriy 2018-10-13 19:23:24 UTC
Created attachment 373920 [details]
gstreamer.log.xz.part-3
Comment 14 Dmitriy 2018-10-13 19:24:06 UTC
Created attachment 373921 [details]
gstreamer.log.xz.part-4
Comment 15 Nicolas Dufresne (ndufresne) 2018-10-13 19:30:45 UTC
So, my notes. We can see that chrome raises the bitrate, and resolution, till it reaches 720p 1.2MB. The where you start having packet lost. Then it reduces the frame rate aggressively before reducing the encoding rate to about 1Mb, after that you have stopped everything.

There is warning about not getting RTCP RR in time, would need investigation. I don't see any trace that would indicate that retransmission is enabled. Does that demo advertise this ? If you aren't using some sort of FEC or transmission, the rate adaptation will always lead to visual corruption. There is not trace that would indicate a keyframe request, if this is broken, obviously this is a problem. I'm not sure how to interpre the hitogram traces.

Histogram: WebRTC.Video.KeyFramesSentInPermille recorded 1 samples, mean = 12,0 (flags = 0x41)

Does this mean it has sent 12 keyframes ? Maybe keyframe at the edge rate just never make it.

Btw, on slow computer, you have packet lost whenever the CPU is too busy to dequeue the packet from the UDP socket, it's not always related to the network capacity. I don't think that chrome algo is really designed to cope with that. Though, you can increase the queue size (see buffer-size on udpsrc), and then sysctl net.core.rmem_max=value to ask the kernel to allow more then the small default. You could also just try and add a queue after udpsrc, not sure how that work with webrtcbin (if it's not already the case). There is already a queue of course, it's the rtpjitterbuffer, but it does not perform very well iirc on loaded systems.

(Haven't looked at the gst logs yet)
Comment 16 Dmitriy 2018-10-19 07:14:06 UTC
(In reply to Nicolas Dufresne (ndufresne) from comment #15)
> There is warning about not getting RTCP RR in time, would need
> investigation. I don't see any trace that would indicate that retransmission
> is enabled. Does that demo advertise this ? If you aren't using some sort of
> FEC or transmission, the rate adaptation will always lead to visual
> corruption. 
Webrtcbin does not create in sdp fec, ulpfec, red in offer/answer.
How to use fec with webrtcbin?

> Btw, on slow computer, you have packet lost whenever the CPU is too busy to
> dequeue the packet from the UDP socket, it's not always related to the
> network capacity. I don't think that chrome algo is really designed to cope
> with that. Though, you can increase the queue size (see buffer-size on
> udpsrc), and then sysctl net.core.rmem_max=value to ask the kernel to allow
> more then the small default. You could also just try and add a queue after
> udpsrc, not sure how that work with webrtcbin (if it's not already the
> case). There is already a queue of course, it's the rtpjitterbuffer, but it
> does not perform very well iirc on loaded systems.
Even if you adjust the parameters of the kernel, even though it will bear fruit, but this is not a solution. Packet loss is inevitable. Also, the gstreamer can be run on a machine on which there is no possibility to change the parameters of the kernel.

If a lot of independent video streams are launched on one machine and the machine for some reason or other slows down, the image will broken in all streams at once. And it will not depend on the characteristics of the processor. No matter how powerful it may be, the problem will be!
Comment 17 Dmitriy 2018-10-19 07:50:37 UTC
I did the FEC as shown in the example https://github.com/GStreamer/gst-plugins-bad/blob/master/tests/examples/webrtc/webrtctransceiver.c

sdp:
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 nack pli
a=framerate:30
a=rtpmap:97 red/90000
a=rtpmap:98 ulpfec/90000

Problem still exists. The video also pixelated.
Comment 18 Dmitriy 2018-10-31 04:16:02 UTC
Created attachment 374112 [details]
Chrome stats
Comment 19 Dmitriy 2018-10-31 04:27:38 UTC
I was on vacation, so it was not possible to experiment.

I tried to configure the linux kernel, but that did not change anything.

Made more convenient output of data on packet loss from get-stats.

Video pixelated immediately after the first loss. Next, the packet lost counter increases by about 100 per second. This goes on for a few minutes. Then the video is restored and the packet lost counter stops increasing. After a few seconds, everything repeats.

Changing net.core.rmem_max gives nothing.
sysctl -w net.core.rmem_max=16777216

I compared the data from the get-stats with the data from /proc/net/udp.

# cat /proc/net/udp | grep ':DDED'
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops
 1060: 8201A8C0:DDED 00000000:0000 07 00000000:00000000 00:00000000 00000000  1000        0 1202334542 2 000000009ee2ba36 10603

From get-stats:
Inbound RTP Stats:
        ssrc: 1183057958
        fir-count: 0
        pli-count: 0
        nack-count: 0
        packets-received: 791085
        bytes-received: 917999656
        jitter: 1
        packets-lost: 98057


/proc/net/udp in 10 times less. Packet lost counter increases by about 100 per second, but value drops in /proc/net/udp changes extremely rarely, no correlation.

All of the above said about CPU Intel Core i5-2300 and chrome <-> gstreamer without vaapi.
On CPU Intel Core i7-7500U in /proc/net/udp always 0, but after ~10 minute video pixelated, get-stats show packets-lost ~ 100 in sec and /proc/net/udp everything is 0.

Why ~20 sec no packet drops and then packet lost regularly?

The chrome graphics show that it does not send huge frames. Nack PLI not working. Why decodebin not send event GstForceKeyUnit?
How to make it work?
Comment 20 GStreamer system administrator 2018-11-03 14:36:08 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/799.