Created attachment 359399 [details] [review]
kms: set buffer to NULL if allocation failed

If bo allocation failed we destroy the buffer and return GST_FLOW_ERROR,
but the @buffer pointer was still pointing to the address of the
destroyed buffer. gst_kms_sink_copy_to_dumb_buffer() was then trying to
unref it when bailing out causing a crash.

Best to return NULL rather than an invalid pointer if the allocation
failed.

Review of attachment 359399 [details] [review]:

gst_buffer_new() will abort on failure. I'd say, remove the check and move the bo allocation up.

Review of attachment 359399 [details] [review]:

While at it, can you fix the other code. gst_kms_sink_copy_to_dumb_buffer() does not initialize buf, and assuming that gst_buffer_pool_acquire_buffer() have set it to NULL is wrong there (just like assuming it was left untouched, but this is cleaner to do so of course.

Created attachment 359400 [details] [review]
kms: fix crash if bo allocation failed when copying to dumb buffer

If bo allocation failed we destroy the buffer and return GST_FLOW_ERROR,
but the @buffer pointer was still pointing to the address of the
destroyed buffer. gst_kms_sink_copy_to_dumb_buffer() was then trying to
unref it when bailing out causing a crash.

Leave @buffer untouched if allocation failed to fix the crash.
Also remove the check on *buffer being not NULL as gst_buffer_new()
will abort if it failed.

Attachment 359400 [details] pushed as 7e38fb5 - kms: fix crash if bo allocation failed when copying to dumb buffer

Also in 1.12, 2b8edd9c0f23e1aedc68fd2271931bce707d2ac4