GNOME Bugzilla – Bug 656923
[playbin2] Improve decoder selection by considering formats supported by the sink
Last modified: 2011-10-03 08:24:53 UTC
Previously playbin2 was using any discovered sink element while autoplugging. This made it possible for formats not supported by the traditional video sinks to provide their own video sink and work seamlessly in applications like Totem. In our case this was useful for hardware acceleration support for video where a decoder could expose a specific kind of caps and its own video sink to handle them. This broke since a commit from Sebastian on March 16th 2011 which made playbin2 reject discovered sink if the configured sink (through the video-sink property) was not able to handle that type. I fail to see the reason for removing that feature from the commit messages and this is quite a problem for us as this breaks video acceleration in Totem. Could we revert that change or get some explanation as to why it's necessary ? Thanks in advance, Julien
The explanation is in the bug report referenced in the commit message, Bug #632788. It's not clear to me exactly what problem you're describing. Could you give a specific example?
I got confused by git blame and could not find the proper commit message then, sorry about that. Before when decodebin2 was discovering a sink, playbin2 was remembering that sink element and was using it instead of the one provided by the application. This was annoying in some sense because if you provided a sink element and were expecting to set properties on it you had to make sure it was actually the one that was in use. For example when decoding H264 video decodebin2 was autoplugging qtdemux then Fluendo VA decoder which generated video/x-fluendo-va output caps and then the Fluendo VA sink which handled those caps. Playbin2 was then using the Fluendo VA sink and everything went fine. For non accelerated video, no sink were encountered and the configured video sink was used. That's how we've been supporting transparent hardware acceleration for Totem on various distro for the past 2 years. With that change playbin2 will discover the Fluendo VA sink and will then check if the sink configured by the application can handle the caps. If it can then it will expose the pad and if it can't it will skip that sink and try to find another element that could convert the caps in something that can be handled by the configured sink. In our case, no other element can handle video/x-fluendo-va and the configured sink in the case of Totem is most likely ximagesink which can't handle anything but RGB. So we end up with no video showing up in Totem. I understand that in GStreamer POV this can be easily fixed by not configuring any video sink on playbin2 and let it do its magic. But the thing is that Totem is not programed this way so it will set gconfvideosink and fallback to ximagesink. Our current approach of working around this is to make a fluvaautosink element that will be selected by autovideosink and which will handle the switch between fluvasink and xvimagesink depending on the incoming caps. That works but I m not sure that the new behavior is actually "better".
Ok, that's what I was thinking. This looks like a regression. IIRC, totem no longer uses gconfvideosink, but I don't think this significantly affects the analysis.
Indeed, in current git master, Totem seems to be using clutter for video rendering. This won't make it easy for us either. I need to talk with Bastien about that before this lands in public distro releases.
(In reply to comment #4) > Indeed, in current git master, Totem seems to be using clutter for video > rendering. This won't make it easy for us either. I need to talk with Bastien > about that before this lands in public distro releases. Probably worth a separate discussion, but the reason for totem switching to cluttervideosink is that certain app want to overlay the video with UI elements at some point. This is useful for image/video capture apps as well.
(In reply to comment #5) > Probably worth a separate discussion, but the reason for totem switching to > cluttervideosink is that certain app want to overlay the video with UI elements > at some point. This is useful for image/video capture apps as well. I understand the reason perfectly. I am just worried that hardware acceleration will have constraints that will be difficult to accomodate with Clutter. We've already experienced quite some difficulties with customers to integrate VAAPI decoding inside a Clutter surface, this will get harder and harder with every different API.
I understand the reasoning behind the move to cluttervideosink. However, the move to inexpensive x86 based PC's with small form factors and low power CPUs like the Intel Atom places more emphasis on the performance of the hardware assist provided by the embedded GPU. If only the GPU vendors could standardize on a well defined API.
Module: gst-plugins-base Branch: master Commit: b0b4e286c8cde2e79a959a444a2c68e99c3f29c6 URL: http://cgit.freedesktop.org/gstreamer/gst-plugins-base/commit/?id=b0b4e286c8cde2e79a959a444a2c68e99c3f29c6 Author: Josep Torra <n770galaxy@gmail.com> Date: Thu Sep 15 09:23:54 2011 +0200 playbin2: autoplug sink if stream is incompatible to the configured one Fixes regression since 0.10.33 where sinks that can cope with non raw caps or custom caps are not autoplugged if there's a sink configured with the properties video-sink and audio-sink which cannot handle the stream. This change checks for compatibility on the configured one and use it if success. Otherwhise it tries with the found factories. Fixes the bug.
Thanks all for your quick response to this problem.
Note that this change was reverted again because it causes broken behaviour and breaks newer releases of totem. We agreed that the previous (pre-.35 behaviour) is broken and a bug and the current behaviour is correct, deterministic and allows the application to handle stuff properly while the old behaviour can't be handled properly by applications and just worked in some applications by luck. The solution to the problem that was solved by relying on the old, broken behaviour would be, to make decodebin2/playbin2 more aware of decoders and improve the autoplugging of decoders by considering the caps supported by the sink instead of just using something with the highest rank.
Will this be fixed sometime soon then? We use Totem and hardware accelerated codecs from Fluendo. Totem should at least allow the user to specify the video sink. This would circumvent the problem as the Fluendo hardware accelerated codec would be autoplugged due to rank and the clutter sink would be overridden by the specified Fluendo hardware accelerated video sink.
> Totem should at least allow the user to specify the video sink. You would need to persuade the Totem maintainer of that.
Created attachment 196626 [details] IRC discussion about this change Attaching today's IRC discussion about this issue.
I think, based on the IRC discussion, that the change to make playbin/decodebin a bit more intelligent is worth the effort. The Totem change that I proposed is really kind of nasty and a hack at best. I guess the real problem is the playbin video sink override that allow the auto plugged video sink to be supplanted by the one specified. Gosh, I just really want to take advantage of hardware accelerated elements for decoding and rendering elegantly. We run on a dual core Atom with NVidia hardware assist for decoding and rendering and employ Totem.
A patch for this is in bug #660598 already, closing this one as a duplicate of the bug with the patch. *** This bug has been marked as a duplicate of bug 660598 ***