GNOME Bugzilla – Bug 785531
appsink: forward the Protection Event to the application
Last modified: 2018-11-03 11:58:12 UTC
Send the protection message to the application when we receive the protection Event in appsink.
Created attachment 356519 [details] [review] appsink: forward the protection event to the application
Review of attachment 356519 [details] [review]: I'm not sure having an message here is the best API. Maybe it would be better to add a api in the style of the others so the ordering with the data can be preserved?
Do you need to extract this in the application only once at the beginning, or does it change at runtime and you need to be notified when it changes?
Yes i need to be notified when it changes at runtime. For example in the adaptive streaming, when the video resolution is changed, thus the DRM KeyId may be changed.
(In reply to Olivier Crête from comment #2) > Review of attachment 356519 [details] [review] [review]: > > I'm not sure having an message here is the best API. Maybe it would be > better to add a api in the style of the others so the ordering with the data > can be preserved? Do you mean to use signal? I don't understand what do you mean by 'api in the style of the others'.
Why do you need this information in the application from appsink, and not only inside the decryptor element and the application logic connected to it? What is your application doing based on the event / message? Also Olivier's comment makes sense, the ordering is not preserved here so you might receive a message after a new protection event already arrived in the appsink. It all depends a bit on what you actually want to do with the information. He means an API like pull_sample() that works for serialized events instead.
(In reply to Sebastian Dröge (slomo) from comment #6) > Why do you need this information in the application from appsink, and not > only inside the decryptor element and the application logic connected to it? > What is your application doing based on the event / message? > > Also Olivier's comment makes sense, the ordering is not preserved here so > you might receive a message after a new protection event already arrived in > the appsink. It all depends a bit on what you actually want to do with the > information. > > He means an API like pull_sample() that works for serialized events instead. There is my use case in WebKit MSE/EME implementation: I have two pipelines. Pipeline1 (AppendPipeline): httpsrc ---> demux ---> appsink Pipeline2 (PlaybackPipeline):appsrc --->parser---> decryptor--->decoder--->Sink Application (MSE/EME implementation): Pipeline1 ---> ( Buffer Manager ) ---> Pipeline2 When the Demux detects the encrypted content, it sends a Protection Event that is propagated until appsink in pipeline1. So i have two solutions to send the event to the application : 1. Add the current patch. 2. Send a bus message from demux to the application. What do you suggest ?
Why do you split it up into two different pipelines? There's nothing technically wrong with posting messages from the demuxer, and it would also be required probably (see the qtdemux patch about selecting between multiple DRM systems). Overall this nonetheless looks like you have a misunderstanding about how the protection API is supposed to be used. Basically, everything should be handled by the decryptor element. The demuxer and parser before that extracts the "common" knowledge from the streams and provides it downstream via the buffers/events/metas/caps. It might have to ask the application to do some kind of decision (see above), but that's about it. The decryptor then makes use of this information and does everything that is needed for actually decrypting the stream. This might include asking the application for further information and blocking any processing until that information is available. Such information could involve asking the application to do its stuff to contact any DRM/license servers for example so that decryption can happen, or it sounds like in your case the application is actually responsible for doing decryption ("buffer manager"?) so it would pass the buffers to the application (CDM?), get buffers back from the application and pass them onwards. There should be no reason to split the pipeline for this, and it could all be relatively self-contained in the decryptor element (which then does the communication with the app).
(In reply to Sebastian Dröge (slomo) from comment #8) > Why do you split it up into two different pipelines? > It is the webkit/Gstreamer architecture > There's nothing technically wrong with posting messages from the demuxer, > and it would also be required probably (see the qtdemux patch about > selecting between multiple DRM systems). OK I'll do that > Overall this nonetheless looks like > you have a misunderstanding about how the protection API is supposed to be > used. > > > Basically, everything should be handled by the decryptor element. The > demuxer and parser before that extracts the "common" knowledge from the > streams and provides it downstream via the buffers/events/metas/caps. It > might have to ask the application to do some kind of decision (see above), > but that's about it. The decryptor then makes use of this information and > does everything that is needed for actually decrypting the stream. This > might include asking the application for further information and blocking > any processing until that information is available. Such information could > involve asking the application to do its stuff to contact any DRM/license > servers for example so that decryption can happen, I agree > or it sounds like in your > case the application is actually responsible for doing decryption ("buffer > manager"?) so it would pass the buffers to the application (CDM?), get > buffers back from the application and pass them onwards. No it isn't doing the decryption, the buffer manager (Source Buffer) is a part of the application, responsible for implementing the MSE (Media Source Extension) algorithmic, regardless of media platform, Gstreamer or others The First pipeline (Append Pipeline) is responsible to puting the content in a packets (GstBuffers) with a metadatas like timestamp, duration and size, that are needed for MediaSource implementation (Buffer Manager). > > There should be no reason to split the pipeline for this, and it could all > be relatively self-contained in the decryptor element (which then does the > communication with the app). I'll remove the current patch and I'll add a mechanism in Demux (MatroskaDemux and Qtdemux ) to send a message on the bus when detecting encrypted content or any change in encryption metadata like a new KeyID in run time. What do you think ?
> > or it sounds like in your > > case the application is actually responsible for doing decryption ("buffer > > manager"?) so it would pass the buffers to the application (CDM?), get > > buffers back from the application and pass them onwards. > > No it isn't doing the decryption, the buffer manager (Source Buffer) is a > part of the application, responsible for implementing the MSE (Media Source > Extension) algorithmic, regardless of media platform, Gstreamer or others > > The First pipeline (Append Pipeline) is responsible to puting the content in > a packets (GstBuffers) with a metadatas like timestamp, duration and size, > that are needed for MediaSource implementation (Buffer Manager). Ah right, I remember now :) Thanks for refreshing my memory. That perfectly makes sense then. > > There should be no reason to split the pipeline for this, and it could all > > be relatively self-contained in the decryptor element (which then does the > > communication with the app). > > I'll remove the current patch and I'll add a mechanism in Demux > (MatroskaDemux and Qtdemux ) to send a message on the bus when detecting > encrypted content or any change in encryption metadata like a new KeyID in > run time. > > What do you think ? Why do you need this information from the demuxer and not from the decryptor? What would the application do with this information? A new key-id at least seems like something the decryptor should only care about, and then request whatever else it needs for that new key-id to do decryption again.
> > I'll remove the current patch and I'll add a mechanism in Demux > > (MatroskaDemux and Qtdemux ) to send a message on the bus when detecting > > encrypted content or any change in encryption metadata like a new KeyID in > > run time. > > > > What do you think ? > > Why do you need this information from the demuxer and not from the > decryptor? What would the application do with this information? > > A new key-id at least seems like something the decryptor should only care > about, and then request whatever else it needs for that new key-id to do > decryption again. If I send the information from the decryptor, I must stop the playback in order to let the application get the appropriate license. I need the information from the demuxer in the goal to anticipate the license acquisition and no interrupt, if possible, the playback The demuxer is in the AppendPipeline and the Decryptor is in the PlaybackPipeline. AppendPipeline: httpsrc ---> demux ---> appsink PlaybackPipeline: appsrc --->parser---> decryptor--->decoder--->Sink AppendPipeline ---> ( Application ) ---> PlaybackPipeline Note: The Decryptor is in the PlaybackPipeline because we use SVP (Secure Video Path), so it should be just before the Decoder, and we don't put it in the AppendPipeline because we can't store in the application queue a decrypted content in SVP that is limited in size.
Ah I see, so you want to get that info ASAP and prevent interruptions, and the pipeline setup is like that due to legal reasons. Posting it from the demuxer is probably fine, yes. Note however that you then need to handle annoying cases like the decryptor using an old key while you already have a new one, and then you request yet another one. Your key retrieval server needs to support giving out many keys at the same time then.
(In reply to Sebastian Dröge (slomo) from comment #12) > Ah I see, so you want to get that info ASAP and prevent interruptions, and > the pipeline setup is like that due to legal reasons. Not only for legal reasons, it is because WebCore in WebKit needs to handle the buffer ranges, etc. That's why the the pipeline is decoupled into two. > Posting it from the demuxer is probably fine, yes. Note however that you > then need to handle annoying cases like the decryptor using an old key while > you already have a new one, and then you request yet another one. Your key > retrieval server needs to support giving out many keys at the same time then. IMHO, the way to go is a message from the appsink because you can hook to it in the mainloop or the element thread. Anyway, for now we either go with this solution in WebKit or we just add a probe in the sink pad of the appsink filtering those messages up to the bus.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/issues/369.