I can reproduce this.

it works fine here. 

This error is supposed to happen like this:
 - buffer arrives in sink (with new caps likely)
 - buffer is not from the imagesink pool.
 - sink allocates an image to render the buffer into, taking the caps of the
   buffer for determining the size of the image.
 - sink find that this new image is smaller than the input buffer and posts
   this error.

Either the input buffer size or the sink image size is wrong.  Enabling the ximagesink debug info will reveal the ximage size. There is no easy debug to enable to find the size of the input buffer.

I get some debug info like this (I added some stuff to print out more info about the current caps):

0:00:02.562920155 15254  0x82d55a8 LOG             ximagesink ximagesink.c:1668:gst_ximagesink_buffer_alloc:<ximagesink0> a buffer of 174080 bytes was requested with caps video/x-raw-rgb, bpp=(int)32, depth=(int)24, endianness=(int)4321, red_mask=(int)65280, green_mask=(int)16711680, blue_mask=(int)-16777216, width=(int)320, height=(int)136, framerate=(fraction)2997/125, pixel-aspect-ratio=(fraction)1/1 and offset 18446744073709551615
0:00:02.563074730 15254  0x82d55a8 LOG             ximagesink ximagesink.c:1706:gst_ximagesink_buffer_alloc:<ximagesink0> trying to resize to window geometry ignoring aspect ratio
0:00:02.563090992 15254  0x82d55a8 DEBUG           ximagesink ximagesink.c:1725:gst_ximagesink_buffer_alloc: we would love to receive a 470x268 video
0:00:02.567073607 15254  0x82d55a8 DEBUG           ximagesink ximagesink.c:1757:gst_ximagesink_buffer_alloc: peer pad accepts our desired caps video/x-raw-rgb, bpp=(int)32, depth=(int)24, endianness=(int)4321, red_mask=(int)65280, green_mask=(int)16711680, blue_mask=(int)-16777216, width=(int)470, height=(int)268, framerate=(fraction)2997/125, pixel-aspect-ratio=(fraction)1/1 buffer size is now 503840 bytes
0:00:02.567118170 15254  0x82d55a8 DEBUG           ximagesink ximagesink.c:220:gst_ximage_buffer_finalize:<ximagesink0> destroy image 0xb4f62888 as its size changed -1x-1 vs current 320x136
0:00:02.567161804 15254  0x8187460 LOG             ximagesink ximagesink.c:743:gst_ximagesink_ximage_put:<ximagesink0> XShmPutImage on 0xb4f62810, src: 0, 0 - dest: 75, 66, dim: 320x136, win 470x268
0:00:02.617312423 15254  0x82d55a8 DEBUG           ximagesink ximagesink.c:427:gst_ximagesink_ximage_new:<ximagesink0> creating image 0xb4f62900 (470x268)
0:00:02.617338153 15254  0x82d55a8 LOG             ximagesink ximagesink.c:459:gst_ximagesink_ximage_new:<ximagesink0> XShm image size is 503840, width 470, stride 1880
0:00:02.623394736 15254  0x82d55a8 DEBUG           ximagesink ximagesink.c:220:gst_ximage_buffer_finalize:<ximagesink0> destroy image 0xb4f62900 as its size changed 470x268 vs current 320x136
0:00:02.625073642 15254  0x82d55a8 LOG             ximagesink ximagesink.c:1598:gst_ximagesink_show_frame:<ximagesink0> normal buffer, copying from it
0:00:02.625089393 15254  0x82d55a8 DEBUG           ximagesink ximagesink.c:1600:gst_ximagesink_show_frame:<ximagesink0> creating our ximage
0:00:02.625102274 15254  0x82d55a8 DEBUG           ximagesink ximagesink.c:427:gst_ximagesink_ximage_new:<ximagesink0> creating image 0xb4f62978 (320x136)
0:00:02.625114284 15254  0x82d55a8 LOG             ximagesink ximagesink.c:459:gst_ximagesink_ximage_new:<ximagesink0> XShm image size is 174080, width 320, stride 1280
0:00:02.625713786 15254  0x82d55a8 WARN            ximagesink ximagesink.c:1608:gst_ximagesink_show_frame:<ximagesink0> Buffer size 503840 (320x136 pixels), ximage size 174080, caps video/x-raw-rgb, bpp=(int)32, depth=(int)24, endianness=(int)4321, red_mask=(int)65280, green_mask=(int)16711680, blue_mask=(int)-16777216, width=(int)320, height=(int)136, framerate=(fraction)2997/125, pixel-aspect-ratio=(fraction)1/1
0:00:02.625732252 15254  0x82d55a8 WARN            ximagesink ximagesink.c:1612:gst_ximagesink_show_frame:<ximagesink0> error: Failed to create output image buffer of 320x136 pixels
0:00:02.625741937 15254  0x82d55a8 WARN            ximagesink ximagesink.c:1612:gst_ximagesink_show_frame:<ximagesink0> error: XServer allocated buffer size did not match input buffer
ERROR: from element /pipeline0/ximagesink0: Failed to create output image buffer of 320x136 pixels
Additional debug info:
ximagesink.c(1612): gst_ximagesink_show_frame (): /pipeline0/ximagesink0:
XServer allocated buffer size did not match input buffer
Execution ended after 2061481499 ns.
Setting pipeline to PAUSED ...
0:00:02.626066428 15254  0x82d55a8 DEBUG           ximagesink ximagesink.c:1631:gst_ximagesink_show_frame: could not create image
Setting pipeline to READY ...
Setting pipeline to NULL ...
0:00:02.694799818 15254  0x804e470 DEBUG           ximagesink ximagesink.c:212:gst_ximage_buffer_finalize:<ximagesink0> destroy image 0xb4f62810 because the sink is shutting down
FREEING pipeline ...

Created attachment 116511 [details]
transform,ximage log of this with some extra instrumentation

I see the error, the output size field is overwritten with a wrong value. fix and unit test will follow soon.

This fixes it for me:

        * libs/gst/base/gstbasetransform.c:
        (gst_base_transform_transform_caps),
        (gst_base_transform_prepare_output_buffer),
        (gst_base_transform_buffer_alloc), (gst_base_transform_suggest):
        Don't overwrite the outsize when calculating the expected size of a new
        buffer because we still need it in case we cannot process the new
        buffer.
        When converting the size of the new buffer to an upstream size, actually
        use the expected size of the buffer, not some other random value.
        Use an atomic int to signal that a new upstream caps suggestion is
        available.
        When we can convert the current buffer to a new format, check if the
        buffer size is of the expected size and allocate a new buffer of the
        expected size when this is not the case.

        * tests/check/libs/transform1.c: (GST_START_TEST):
        remove ifdeffed code from the unit test.