GNOME Bugzilla – Bug 664733
queue: special support for encoded streams
Last modified: 2018-05-01 08:38:48 UTC
we need a queue for encoded streams ie streams with I-frames and other (P,B) frames. so we developed one such elements. we modify queue, but we can probably can do it to inherit a new element from queue. do you interested about it? ie. do we make the source better and send it?
What exactly is your new queue doing different than the current queue? Maybe just attach your changes and then we'll see if this is something useful in general :)
currently queue can drop all/any frames which is not suitable for any encoded (h264, mpeg4, etc.) video streams. since in this case if you drop an I-frame you must also drop all consequent P and B frames until the next I-frame. So we plane to add a new properties: encoded stream true/false default: false to queue. after that queue works before, but if this properties set to true, the: - if drop a frame thne drop all consequent frames until the next I-frame, - which means all stream produced by the queue always can be decoded and never gives degraded video stream.
So this is for the leaky mode of queue(1)? Sure, a patch for that would make sense but the leaky mode is really just a hack anyway in all cases I saw so far
we think leaky mode is really useful. eg when there is a network stream and both the server and client can be slower then the others mainly with encoded video stream. in this case a leaky queue which able to handle encoded stream is very useful. we finally able to write such a patch which can handle (and auto detect) both raw and encoded stream and do the frame dropping in the proper way. on monday we'll send a patch to queue. another interesting feature would be to combine queue and valve into one element. unfortunately in encoded stream case it's not possible to (ie. not working properly) if we simple put together a queue and a valve since the frame dropping and the frame type (ie. I, P or B) should have to combine and it'd be best if we can add this properties to queue too.
Created attachment 202360 [details] [review] patch for frame type
Created attachment 202361 [details] [review] add a possibility to stop streamering
the first patch gstreamer-delta.patch add a possibility to examine the frame time in and encoded stream and always drop frame in leaky mode in a way that the resulted stream are correctly decodable (if the input stream was correct). by default it's working as before. if you would like to use this feature you'd have to set the "check-frame-type" properties to true. the second patch add a possibility to stop the streaming after the queue, but still buffering in the queue the incoming frames. it can be also controllable through a properties.
Or you could collect a whole GOP in a buffer list, and then let the queue handle/drop the whole buffer list (as queue2 in git does already), instead of adding specialist knowledge to the queue. Has different latency characteristics though.
does the queue2 can drop frames? ie. is queue2 has leaky mode as queue has? if not then could you port this feature from queue2 to queue? and what does the "different latency characteristics" means?
Created attachment 202375 [details] [review] patch for frame type
any change that these patches accepted before 10.36?
Not really.
ok then which kind of patch would be accepted in the next release or we use our custom build of queue which take care of encoded frames?
is there any change to be such patch will be accepted or should we use our local patched version?
I don't think we have made up our minds yet if we want this kind of feature in queue or not, but the patches certainly won't go in in their current form, and probably not any time soon either (since most people are focusing either on 0.10 bugfixing for the release or on 0.11), so you should probably use your own version for now. You might want to fix up things like !(GST_BUFFER_FLAGS(GST_BUFFER_CAST (leak)) && GST_BUFFER_FLAG_DELTA_UNIT) though, which probably doesn't do what you want it to do. Also, please run gst-indent over the .c files before submitting patches, and ideally submit them in git format-patch format (the commit hook will check the indentation automatically then).
(In reply to comment #15) > You might want to fix up things like > > !(GST_BUFFER_FLAGS(GST_BUFFER_CAST (leak)) && GST_BUFFER_FLAG_DELTA_UNIT) > > though, which probably doesn't do what you want it to do. Also, please run we though it;s not a bug, since we check in this way whether the next frame in the queue is delta or not. and if it's we drop it. > gst-indent over the .c files before submitting patches, and ideally submit them > in git format-patch format (the commit hook will check the indentation > automatically then). we'll update both patch to a git format-patch format tomorrow.
let we explain these two patches. 1. the streaming properties. this is something like a combination of leaky queue and a valve. this means you can stop the streaming at the src pad without dropping the frames but of course if the queue is full than start to drop. at the src pad queue do not give frames if the queue was empty, now we modify it not to give also if the streaming off. 2. the check-frame-type properties. in leaky mode the produced frames at the src pad always decodable even if the flow contains I, P or B frames. there are two cases: a. leaky mode is upstream: if we drop an incoming frame, then we drop all new frames until it's a I-frame (since otherwise the stream won't be decodable). b. leaky mode is downstream: if we drop a frame at the end of the internal queue we drop all previous frames until next I-frame, but don't drop events. if the queue becomes empty after this, then we also drop all incoming new frames until the first I-frame. i hope it's clear the idea behind these two patches.
Reopening as the question in comment #1 has been answered.
What should we do about this? At least the implementation has to be improved quite a bit, but do we really want this feature or should it be implemented elsewhere (e.g. bufferlist per GOP)?
IMO, if you think leaky behaviour is interesting and if you drop a non-delta frame, I think it makes sense to drop the following delta-frames until you see a non-delta frame again.
that's exactly what we do. anyway we use this patch in production since it's a very important feature to be able to protect any component in a networked environment ie. the servers which may produce too many video frames which can cause out of memory on the server side eg when you see a live stream it's more important to see the actual images rather then all frames. and o the client side eg. when the display is slower then the produces (network/anything else) to render the video eg. on a 100 parallel video wall stream case it's a rather larger cpu/gpu strees and in this case it's very useful to drop frames in the decoded stream (since we also save cpu compared to if we drop after the decoding). i can send out current patches but only for the 0.10 series since we still use it (until we manage the rather new glib requirement of gst-1.0).
No updates in 5 years. Closing. Also dropping "late" frames can be done in different ways by media-specific elements (and not queue which should remain media-agnostic, and the leaky behaviour is really not something you should rely on).
can you tell me any such media-specific elements?