After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 722666 - avoid decoding not-linked streams
avoid decoding not-linked streams
Status: RESOLVED OBSOLETE
Product: GStreamer
Classification: Platform
Component: gstreamer (core)
git master
Other All
: Normal enhancement
: git master
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2014-01-21 01:20 UTC by Thiago Sousa Santos
Modified: 2018-11-03 12:19 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description Thiago Sousa Santos 2014-01-21 01:20:38 UTC
Since the 0.10 -> 1.0 move, the feature that was used to make not-linked streams skip decoding until linked again is gone.

It was done by input-selector returning not-linked on pad-allocs and the decoder would drop that buffer and return not-linked. Nothing has replaced this behavior in 1.0 (yet).

This can be easily demonstrated with the following scenario:

1) Create a test clip with multiple videos:
gst-launch-1.0 videotestsrc num-buffers=500 pattern=0 ! x264enc ! qtmux name=m ! filesink location=3video.mov videotestsrc num-buffers=500 pattern=1 ! x264enc ! m. videotestsrc num-buffers=500 pattern=18 ! x264enc ! m.

2) Play the clip and look at the debug logs for the video decoders:
GST_DEBUG=*videodec*:9,*avdec*:9 gst-launch-1.0 playbin uri=file:///path/to/3video.mov


A similar scenario is solved in dash/mss demuxers using the not-linked return and reconfigure events to stop/reactivate those streams. The use case is a bit different but I guess the mechanism can work the same.

Possible issues:
1) Is this a behavior break?
2) How to avoid dropping buffers that are ahead of the current play position? It would be better if the decoder would only drop a buffer to avoid decoding when it knew that this buffer is too late to be used.
3) How does this interfere with the buffer pools and allocation queries?
Comment 1 Sebastian Dröge (slomo) 2014-01-21 09:01:48 UTC
It should be possible to do this the same way somehow, just 2) seems a potential problem. Maybe we need the position in the reconfigure event? OTOH we would never be too far ahead of the current play position because of a) multiqueue and b) probably a sensible interleaving of audio/video in the container format.
Comment 2 Sebastian Dröge (slomo) 2014-01-21 09:04:35 UTC
There's also another bug about this somewhere already but I can't find it :)
Comment 3 Wim Taymans 2014-01-21 10:44:43 UTC
Maybe for now (until we settle on a good stream-selection API) you can stop pushing on pads not returning GST_FLOW_OK and retry to push when you get a reconfigure event on them?
Comment 4 Thiago Sousa Santos 2014-01-23 15:02:06 UTC
(In reply to comment #1)
> It should be possible to do this the same way somehow, just 2) seems a
> potential problem. Maybe we need the position in the reconfigure event? OTOH we
> would never be too far ahead of the current play position because of a)
> multiqueue and b) probably a sensible interleaving of audio/video in the
> container format.

The problem with this approach is when the content isn't coming from an interleaved source, like adaptive streams or maybe network streams with separate video/audio sources. So it would advance ahead faster than running time dropping all data.
Comment 5 Sebastian Dröge (slomo) 2014-01-23 15:08:21 UTC
multiqueue "fixes" that interleaving
Comment 6 Thiago Sousa Santos 2014-02-17 15:48:29 UTC
Took a deeper look at the base decoder classes and I'm wondering if we can unconditionally drop input buffers without parsing/decoding without breaking stream buffer alignment.

Can we assume that if the segment.format is TIME the input is properly aligned? Or only if parsed/framed=true is present on caps?
Comment 7 Olivier Crête 2014-02-17 16:59:26 UTC
Only if parsed/framed, there are cases where you can get TIME segments without proper alignment, for example, when stuff comes out of a mpegts container or if you send raw MPEG elementary streams over UDP (I've seen people do it).
Comment 8 Nicolas Dufresne (ndufresne) 2014-02-17 17:22:53 UTC
Note that from decoder base class, you would expect if not packetized, to call the parser virtual method. A decision to keep or drop frame should be made after the parse virtual method has been called. Note saying this is THE solution, I have no idea atm in fact.
Comment 9 Arnaud Vrac 2015-05-30 19:25:49 UTC
Hello, I'm curious has any work been done on this ? It's annoying that with a playbin based pipeline without the video or audio flag, a decoder is still used for the disabled type. This might be problematic when using accelerated decoders with limited resources.
Comment 10 Edward Hervey 2015-06-01 09:21:14 UTC
I'm currently looking at this issue (and related stream discovery/selection).

One way to solve this is to:
1) have multiqueue take parsed/packetized streams (i.e. if parsers are needed, put them before)
2) have multiqueue do extra buffering for inactive streams (keep an extra second, or from last keyframe for video)

When the stream is not active => no data is sent
When the stream gets activated => push all pending data

Will update once I have more findings.
Comment 11 GStreamer system administrator 2018-11-03 12:19:42 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/49.