I've just checked and this issue is still extant for gstreamer 1.1.2 in git.

Is there any chance of someone looking into it? X image capture is quite slow at the moment and I was hoping that a working damage system would significantly increase perfomance.

The problem is that, while damage allows to limit the throughput from ximage regarding updates (smaller data to read, you only get updates) ... it reconstructs the images straight away in the source (memcpy of the whole final size, potentially very slow)... and it does that in the same thread as the capture. That results in missing capture of the earliest next frame.

Ideally that reconstruction should happen later (when the data is actually being used) and the source should just send out buffers with the updated region (and a reference to the previous frame).

With 1.x, ximagesrc could create a new type of GstMemory for that:

* Store reference to previous image (or to none if it's a full image or has been reconstructed)
* When mapping (i.e. when someone wants to read it), reconstruct the full output image (based on on damage data + previous image, which might end up being recursive btw).

Maybe a bufferpool would also help with that also (keeping around common/last image).

This would allow getting fast acquisition (from X) and allow delayed reconstruction.

(In reply to comment #2)

> With 1.x, ximagesrc could create a new type of GstMemory for that:
> 
> * Store reference to previous image (or to none if it's a full image or has
> been reconstructed)
> * When mapping (i.e. when someone wants to read it), reconstruct the full
> output image (based on on damage data + previous image, which might end up
> being recursive btw).
> 
> Maybe a bufferpool would also help with that also (keeping around common/last
> image).
> 
> This would allow getting fast acquisition (from X) and allow delayed
> reconstruction.

This sounds like a good design, however I have a use-case where I'd like to convert the frames with damage data into jpegs which are transparent except for the changed data. That way I can simply overlay them on top of each other at the display end, and never have to reconstitute the images in memory. Although, I suppose it would make sense if every Nth frame were a full frame, for ease of recovery in case of loss of a frame.

I *may* be able to convince my boss to give me time to help out on this, as we have a definite use case for it. I'll have to ask.

(In reply to comment #3)

> 
> I *may* be able to convince my boss to give me time to help out on this, as we
> have a definite use case for it. I'll have to ask.

My Boss has, indeed, given me a green light to help out on this. However, I'm afraid my knowledge of gstreamer's memory architecture is scanty, although I'm a quickly learner. Feel free to contact me to coordinate work on this.

I experienced this problem too, maybe use-damage should be set to false by default until this issue is sorted out, what do you think?

Thanks,
   Antonio

(In reply to comment #3)
> This sounds like a good design, however I have a use-case where I'd like to
> convert the frames with damage data into jpegs which are transparent except for
> the changed data. That way I can simply overlay them on top of each other at
> the display end, and never have to reconstitute the images in memory. Although,
> I suppose it would make sense if every Nth frame were a full frame, for ease of
> recovery in case of loss of a frame.

At the top of my head :

Indeed, you should be able to handle both usages (one for plugins who are not aware or care about damaged region and want whole (reconstructed) frames, and the other one for plugins who could make a more efficient usage of the damaged regions).

There are two parts to it:
* The actual data storage (i.e. custom GstMemory)
* The metadata about those damaged regions

In order for this system to work, you'll need a common (shared amongst all buffers/memory of a stream) structure to contain the last fully-reconstructed image/GstMemory. This could/should be stored in a GstBufferPool imho.

For the "damaged" GstMemory you want to know (and privately remember):
* Whether it is a whole or to-be-reconstructed image
* If to-be-reconstructed, a reference to the previous GstMemory (which will chain all the way to the last fully reconstructed image).
* a list of
** GstMemory for this image's partial memory (those are not stored in GstBuffer, otherwise the default gst_buffer_map would give you garbage).
** the x,y pixel offset of that memory in the whole frame
** the width,height pixel offset of that memory in the whole frame
* Implementations of map/unmap vmethod to reconstruct if needed (i.e. the fallback solution). Those would then chain-up and reconstruct a full video image (which further buffers/memory will then use).

That would take care of the "naive" delegated reconstruction (any plugin downstream wanting to map the memory will end up with a full image).

You'll also need to create a GstMeta to provide information on the "damaged" content, for the plugins that can handle it
* Glist of damage information
  * x,y position in pixels
  * width, height in pixels
  * GstMemory reference of the individual damaged area

That would allow plugins that know how to deal with damaged images to be able to get the individual regions.

You would then need to add support to your encoder to create those "transparent" images you mention.

The x,y,width,height would all be in pixels and you'd have to check the GstVideoMeta for the actual type and compute the bytes offsets/strides

Hope this helps

*** Bug 710323 has been marked as a duplicate of this bug. ***

Keeping this open. Has useful information regarding a backwards-compatible and transparent way of handling this use-case.

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/82.