GNOME Bugzilla – Bug 595614
theora/enc/dec playback drops buffers/freezes intermittently
Last modified: 2018-11-03 11:15:41 UTC
Running either of these pipelines leads to choppy playback and frames freezing: gst-launch -v v4l2src ! video/x-raw-yuv, width=640, height=480, framerate=30000/1001 ! theoraenc ! theoradec ! queue max-size-buffers=3 ! xvimagesink This one only freezes at the beginning of playback: gst-launch -v videotestsrc is-live=true ! video/x-raw-yuv, width=640, height=480, framerate=30000/1001 ! theoraenc ! theoradec ! queue max-size-buffers=3 ! xvimagesink this gets worse as the quality setting is increased. on #gstreamer, thaytan suggested that theoraenc/dec weren't handling the latency query event properly as setting sync=false on xvimagesink yields smoother playback.
Is your CPU fast enough to do a real-time encode at that size? If not, you'll get frame dropping, which will end up looking choppy for a variety of reasons.
i think so, it's an intel quad core @ 2.40 ghz.
I confirm the issue here with the first pipeline, it freezes after the first or second frame and prints tons of warnings like: 0:00:01.198691804 2773 0x8e63c78 WARN theoradec theoradec.c:1430:theora_handle_data_packet:<theoradec0> dropping frame because of QoS It works pretty fine if I set sync=false in xvimagesink (no noticeable speed loss nor dropped frames). I'm aware of at least another guy who had the same issue some time ago, CC-ing him. The issue doesn't seem to happen with videotestsrc (maybe it's not the same issue of the reporter). Could it be related to my webcams outputting a couple of broken frames at the beginning (bug 587174)?
So the problem are probably broken timestamps from the v4l2src, which then trigger QoS and framedropping. Bug #595825 might help here
Not sure if bug #595825 is releated. There are two issues with the timestamping in v4l2 I am aware of: 1.) basetime is taken too early 2.) there is jitter in the timestamps as we stamp when we dequeue from userspace The enhancement request I filed (and will work on) is related to the 2nd.
This does not seem to be due to v4l2 timestamps, but simply the extra latency from Theora encoding. On my machine (Intel i7 at... 2.7 GHz or 1.2 GHz, not sure if it speeds up when encoding, but it doesn't really matter), libtheora encodes a 640x480 frame in roughly: default parameters: 0.1 seconds quality=1 speed-level=4: 0.015 seconds The second one yields fluid motion, albeit with bad quality. The first one, well... that's the same latency as 3 whole frames. Compare with the whole pipeline latency which was set to 0.03 - a third of the latency introduced by theoraenc. Should elements such as encoders estimate and add their own latency to a latency query ? Those don't seem trivial to estimate, as latency will depend on frame complexity. Looking arbitrarily: - schroenc does, but it looks like an inherent frame-based latency, not sure - x264enc does not.
I believe the GstVideoEncoder base class now tries to estimate its own latency. However, I can still reproduce the issue with: gst-launch-1.0 -v v4l2src ! 'video/x-raw,width=640,height=480,framerate=30/1' ! queue ! theoraenc ! theoradec ! queue max-size-buffers=3 ! xvimagesink The question is: is it actually a bug? Arguably theoraenc is simply not fit for low latency use cases, at least not in our default configuration. (I have a vague memory of it taking very very long to encode some frames when there's rapid motion involved).
The video encoder base class has no code to estimate the latency, it still requires the subclass to tell it about the latency.
Still an issue (with higher resolutions).
Might be related to dyanmic framerate from v4l2src. A solution might be to reborn this old experiment: https://bugzilla.gnome.org/show_bug.cgi?id=675761 If I remember, the main issue whas that it's not a perfect solution, you get some drops before the latency get updated. Also updating the latency will freeze the image for some time, and underflow shortly the audio.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/issues/24.