I think would be really useful to add per frame metadata (or a metadata every n frame) at muxer level, for example on qtmux. This way we can do the following usage:

1) save a video stream and when muxing add some metadata info (for example an encryption key)
2) play the stream and verify this info (maybe post a message on the bus) so the user can be notifyed if the video hasn't the expected key

actually gstreamer can do something like this using metadatamux, but this element works only with jpeg and png image so require cpu intensive transcoding if you have a stream different from mjpeg. Add this to a muxer such as qtmux could give the same result without a cpu intensive transcoding and make the metadata muxing much more general, what do you think about?