After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 664733 - queue: special support for encoded streams
queue: special support for encoded streams
Status: RESOLVED OBSOLETE
Product: GStreamer
Classification: Platform
Component: gstreamer (core)
git master
Other Linux
: Normal enhancement
: git master
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2011-11-24 14:57 UTC by Levente Farkas
Modified: 2018-05-01 08:38 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
patch for frame type (5.92 KB, patch)
2011-11-29 10:46 UTC, Levente Farkas
none Details | Review
add a possibility to stop streamering (3.38 KB, patch)
2011-11-29 10:47 UTC, Levente Farkas
needs-work Details | Review
patch for frame type (5.92 KB, patch)
2011-11-29 13:55 UTC, Levente Farkas
needs-work Details | Review

Description Levente Farkas 2011-11-24 14:57:49 UTC
we need a queue for encoded streams ie streams with I-frames and other (P,B) frames. so we developed one such elements. we modify queue, but we can probably can do it to inherit a new element from queue.
do you interested about it? ie. do we make the source better and send it?
Comment 1 Sebastian Dröge (slomo) 2011-11-25 09:04:24 UTC
What exactly is your new queue doing different than the current queue? Maybe just attach your changes and then we'll see if this is something useful in general :)
Comment 2 Levente Farkas 2011-11-25 12:25:09 UTC
currently queue can drop all/any frames which is not suitable for any encoded (h264, mpeg4, etc.) video streams. since in this case if you drop an I-frame you must also drop all consequent P and B frames until the next I-frame.
So we plane to add a new properties:
encoded stream true/false default: false
to queue. after that queue works before, but if this properties set to true, the:
- if drop a frame thne drop all consequent frames until the next I-frame,
- which means all stream produced by the queue always can be decoded and never gives degraded video stream.
Comment 3 Sebastian Dröge (slomo) 2011-11-25 13:37:53 UTC
So this is for the leaky mode of queue(1)? Sure, a patch for that would make sense but the leaky mode is really just a hack anyway in all cases I saw so far
Comment 4 Levente Farkas 2011-11-25 16:46:44 UTC
we think leaky mode is really useful. eg when there is a network stream and both the server and client can be slower then the others mainly with encoded video stream. in this case a leaky queue which able to handle encoded stream is very useful. we finally able to write such a patch which can handle (and auto detect) both raw and encoded stream and do the frame dropping in the proper way. on monday we'll send a patch to queue.

another interesting feature would be to combine queue and valve into one element. unfortunately in encoded stream case it's not possible to (ie. not working properly) if we simple put together a queue and a valve since the frame dropping and the frame type (ie. I, P or B) should have to combine and it'd be best if we can add this properties to queue too.
Comment 5 Levente Farkas 2011-11-29 10:46:38 UTC
Created attachment 202360 [details] [review]
patch for frame type
Comment 6 Levente Farkas 2011-11-29 10:47:07 UTC
Created attachment 202361 [details] [review]
add a possibility to stop streamering
Comment 7 Levente Farkas 2011-11-29 10:51:16 UTC
the first patch gstreamer-delta.patch add a possibility to examine the frame time in and encoded stream and always drop frame in leaky mode in a way that the resulted stream are correctly decodable (if the input stream was correct). by default it's working as before. if you would like to use this feature you'd have to set the "check-frame-type" properties to true.

the second patch add a possibility to stop the streaming after the queue, but still buffering in the queue the incoming frames. it can be also controllable through a properties.
Comment 8 Tim-Philipp Müller 2011-11-29 10:57:58 UTC
Or you could collect a whole GOP in a buffer list, and then let the queue handle/drop the whole buffer list (as queue2 in git does already), instead of adding specialist knowledge to the queue. Has different latency characteristics though.
Comment 9 Levente Farkas 2011-11-29 11:04:53 UTC
does the queue2 can drop frames? ie. is queue2 has leaky mode as queue has?

if not then could you port this feature from queue2 to queue?

and what does the "different latency characteristics" means?
Comment 10 Levente Farkas 2011-11-29 13:55:06 UTC
Created attachment 202375 [details] [review]
patch for frame type
Comment 11 Levente Farkas 2011-12-11 17:14:35 UTC
any change that these patches accepted before 10.36?
Comment 12 Tim-Philipp Müller 2011-12-11 17:22:23 UTC
Not really.
Comment 13 Levente Farkas 2011-12-11 17:30:27 UTC
ok then which kind of patch would be accepted in the next release or we use our custom build of queue which take care of encoded frames?
Comment 14 Levente Farkas 2012-01-04 13:31:58 UTC
is there any change to be such patch will be accepted or should we use our local patched version?
Comment 15 Tim-Philipp Müller 2012-01-04 13:54:23 UTC
I don't think we have made up our minds yet if we want this kind of feature in queue or not, but the patches certainly won't go in in their current form, and probably not any time soon either (since most people are focusing either on 0.10 bugfixing for the release or on 0.11), so you should probably use your own version for now.

You might want to fix up things like

  !(GST_BUFFER_FLAGS(GST_BUFFER_CAST (leak)) && GST_BUFFER_FLAG_DELTA_UNIT)

though, which probably doesn't do what you want it to do. Also, please run gst-indent over the .c files before submitting patches, and ideally submit them in git format-patch format (the commit hook will check the indentation automatically then).
Comment 16 Levente Farkas 2012-01-04 14:24:29 UTC
(In reply to comment #15)
> You might want to fix up things like
> 
>   !(GST_BUFFER_FLAGS(GST_BUFFER_CAST (leak)) && GST_BUFFER_FLAG_DELTA_UNIT)
> 
> though, which probably doesn't do what you want it to do. Also, please run

we though it;s not a bug, since we check in this way whether the next frame in the queue is delta or not. and if it's we drop it. 

> gst-indent over the .c files before submitting patches, and ideally submit them
> in git format-patch format (the commit hook will check the indentation
> automatically then).

we'll update both patch to a git format-patch format tomorrow.
Comment 17 Levente Farkas 2012-01-13 12:16:32 UTC
let we explain these two patches.

1. the streaming properties.
this is something like a combination of leaky queue and a valve. this means you can stop the streaming at the src pad without dropping the frames but of course if the queue is full than start to drop. 
at the src pad queue do not give frames if the queue was empty, now we modify it not to give also if the streaming off.


2. the check-frame-type properties.
in leaky mode the produced frames at the src pad always decodable even if the flow contains I, P or B frames.
there are two cases:

a. leaky mode is upstream:
if we drop an incoming frame, then we drop all new frames until it's a I-frame (since otherwise the stream won't be decodable).

b. leaky mode is downstream:
if we drop a frame at the end of the internal queue we drop all previous frames until next I-frame, but don't drop events. if the queue becomes empty after this, then we also drop all incoming new frames until the first I-frame.

i hope it's clear the idea behind these two patches.
Comment 18 Tobias Mueller 2012-05-05 13:35:34 UTC
Reopening as the question in comment #1 has been answered.
Comment 19 Sebastian Dröge (slomo) 2013-07-24 08:47:30 UTC
What should we do about this? At least the implementation has to be improved quite a bit, but do we really want this feature or should it be implemented elsewhere (e.g. bufferlist per GOP)?
Comment 20 Wim Taymans 2013-07-24 08:55:24 UTC
IMO, if you think leaky behaviour is interesting and if you drop a non-delta frame, I think it makes sense to drop the following delta-frames until you see a non-delta frame again.
Comment 21 Levente Farkas 2013-07-24 10:41:16 UTC
that's exactly what we do. anyway we use this patch in production since it's a very important feature to be able to protect any component in a networked environment ie. the servers which may produce too many video frames which can cause out of memory on the server side eg when you see a live stream it's more important to see the actual images rather then all frames. and o the client side eg. when the display is slower then the produces (network/anything else) to render the video eg. on a 100 parallel video wall stream case it's a rather larger cpu/gpu strees and in this case it's very useful to drop frames in the decoded stream (since we also save cpu compared to if we drop after the decoding).

i can send out current patches but only for the 0.10 series since we still use it (until we manage the rather new glib requirement of gst-1.0).
Comment 22 Edward Hervey 2018-05-01 07:14:19 UTC
No updates in 5 years. Closing.

Also dropping "late" frames can be done in different ways by media-specific elements (and not queue which should remain media-agnostic, and the leaky behaviour is really not something you should rely on).
Comment 23 Levente Farkas 2018-05-01 08:38:48 UTC
can you tell me any such media-specific elements?