GNOME Bugzilla – Bug 646638
[API] need stream (de)activation API
Last modified: 2016-11-12 14:16:32 UTC
Launching: gst-launch playbin2 uri=file://`pwd`/test.mov flags=0x12 on a (very fast) machine, with test.mov being a full HD video trailer, yields a 30% CPU usage, when decoding just the AAC audio track should use a tenth of that.
That's because disabling video only disables displaying of the video and not the decoding (because you want to be able to enable the video again while staying in PLAYING or PAUSED). With my branch here[0] we can probably implemented this properly by deactivating the video streams upstream at the multiqueues inside decodebin2. [0]http://git.collabora.co.uk/?p=user/slomo/gstreamer.git;a=shortlog;h=refs/heads/stream-active
Unfortunately I don't think these changes can go into 0.10 without breaking something. New 0.11 branches are at http://cgit.collabora.com/git/user/slomo/gstreamer.git/log/?h=activate-stream http://cgit.collabora.com/git/user/slomo/gst-plugins-base.git/log/?h=activate-stream I'll try to add the disable-decoding optimization here later.
Related to https://bugzilla.gnome.org/show_bug.cgi?id=638891, this looks like a good case to use the patch in that bug.
*** Bug 665993 has been marked as a duplicate of this bug. ***
Renaming title so this stream (de)activation is not hidden/buried here. Note that such (de)activation is pretty useful/needed in case where limited resources (think 'embedded') only allow a limited number of codec instances, so you can't have them running on several streams simultaneously. Also adding following comments here since related to multiple streams: * inputselector seems to pass along all queries, including ALLOCATION, so that might lead to confusing and complication for a downstream sink trying to satisfy all that (particularly when it too has limited resources) * at some point in stream switching/activation, maybe inputselector should also send some RECONFIGURE event (upstream) ?
Since we have base audio/video decoder classes now we could have something like this: If a decoder is not linked (via reconfigure/allocation/... notification) then it does not configure itself, but continuously keeps the previous N buffers needed to go back to decoding: * For audio it could be zero (in case of zero latency codecs) to N (in case you need the previous N buffers to decode a buffer). * For video it could be zero (in case of image/I-frame only codecs) to everything from the previous keyframe. When a decoder is notified that it is linked downstream, it then has access to all the data needed to decode and output data immediatly. The base classes could take care of that automatically so that implementations don't have to care whether they are linked or not.
(In reply to comment #5) > Also adding following comments here since related to multiple streams: > * inputselector seems to pass along all queries, including ALLOCATION, so that > might lead to confusing and complication for a downstream sink trying to > satisfy all that (particularly when it too has limited resources) It should only forward for the active stream... > * at some point in stream switching/activation, maybe inputselector should also > send some RECONFIGURE event (upstream) ? ...and send RECONFIGURE events upstream on switching streams so upstream can send new ALLOCATION queries and configure itself (input-selector does this already).
(In reply to comment #6) > * For video it could be zero (in case of image/I-frame only codecs) to > everything from the previous keyframe. Problem here would be crazy streams that only have a single sync point at the very beginning and only dependent frames afterwards for a long, long time.
(In reply to comment #8) > (In reply to comment #6) > > > * For video it could be zero (in case of image/I-frame only codecs) to > > everything from the previous keyframe. > > Problem here would be crazy streams that only have a single sync point at the > very beginning and only dependent frames afterwards for a long, long time. Do you have any example ?
I don't, but it's something that can be created and which we should handle without crashing or filling all available memory
(In reply to comment #10) > I don't, but it's something that can be created and which we should handle > without crashing or filling all available memory Fair enough, but for those we could just establish a limit, and make them decode (instead of piling up memory forever).
Yes, sounds like a good solution :)
Let's discuss this at the hackfest at the end of the month. As I see it, and as has been mentioned above, there are at least three goals here: 1) avoid parsing/decoding of data that's not being used 2) ideally avoid reading and demuxing data that's not going to be decoded or used (where possible, e.g. in qtdemux) 3) make stream switching better/smoother/faster
Something to keep in mind when designing/implementing this is that it should still be possible to have multiple streams of a subtype active at the same time, e.g to have audio descriptions for visually-impaired people mixed with a normal audio track, even if playbin does not support that yet.
*** Bug 570144 has been marked as a duplicate of this bug. ***
*** Bug 645016 has been marked as a duplicate of this bug. ***
Another thing currently missing is api to select a stream by stream-id. Apps that use discoverer and want to select a certain stream will need to connect to playbin::pad-added to build map<stream-id, index> for audio/video/subtitles.
I think many of the things discussed here are now implemented in principle with the new GstStream API and playbin3/decodebin3, so I think I'll close this bug and open new bugs with more narrow scope for specific issues. > Something to keep in mind when designing/implementing this is that it should > still be possible to have multiple streams of a subtype active at the same > time, e.g to have audio descriptions for visually-impaired people mixed with > a normal audio track, even if playbin does not support that yet. The SELECT_STREAMS event API does support that now. > Another thing currently missing is api to select a stream by stream-id. Apps > that use discoverer and want to select a certain stream will need to connect > to playbin::pad-added to build map<stream-id, index> for > audio/video/subtitles. The SELECT_STREAMS event API allows exactly that (takes list of stream IDs). > ... three goals here: > > 1) avoid parsing/decoding of data that's not being used playbin3/decodebin3 will not decode data that's not being used. There may be bugs with selecting and unselecting of streams, but those are implementation bugs and we should file separate bugs for those as we run into them. We currently can't just disable the video in playbin3 for example, it will do it but then it will freeze up shortly after. But from an API/design perspective this is fixed as far as I'm concerned. > 2) ideally avoid reading and demuxing data that's not going to > be decoded or used (where possible, e.g. in qtdemux) Reading/parsing/demuxing. We need to read/demux/parse streams we may want to switch to at short notice. However, we should not read/demux/parse streams that we know are never going to be used. For this we need new stream deactivation API similar to the SELECT_STREAMS event. I will create a new bug for that. > 3) make stream switching better/smoother/faster Should be fixed with playbin3/decodebin3.