Created attachment 334957 [details] [review]
patch to avoid invoking undefined behavior

gdk-pixbuf (from git master, commit 94d739702ea0a7) has the following code in
gdk-pixbuf/gdk-pixbuf.c file.

GdkPixbuf *
gdk_pixbuf_new (GdkColorspace colorspace,
                gboolean      has_alpha,
                int           bits_per_sample,
                int           width,
                int           height)
{
        guchar *buf;
        int channels;
        int rowstride;

        channels = has_alpha ? 4 : 3;
        rowstride = width * channels;
        if (rowstride / channels != width || rowstride + 3 < 0) /* overflow */
                return NULL;
        ...
        buf = g_try_malloc_n (height, rowstride);
        if (!buf)
                return NULL;
}

When this code is compiled with GCC 6.1 or Clang 3.8 with "-O2" flag, the
overflow check (rowstride / channels != width) is eliminated completely by the
compiler. This is because signed integer overflow (rowstride = width *
channels) is undefined and the overflow check occurs after the undefined
behaviour. Large values of the "width" function argument lead to signed integer
overflow and this is not detected at runtime since the overflow check has been
eliminated (optimized away) at compile time. Such large width values result in
a small memory allocation (buf = g_try_malloc_n) and memory corruption problems
in OneLine32 function in io-bmp.c file later on.

See https://bugzilla.gnome.org/show_bug.cgi?id=768738 for details about
the memory corruption problems caused by a malformed BMP file.

We can fix this code by doing the multiplication (rowstride = width * channels) 
operation in unsigned type, instead of signed type. The patch to do so is
attached.

...

The following code pattern (in io-tiff.c) also invokes undefined behaviour and should be fixed,

static GdkPixbuf *
tiff_image_parse (TIFF *tiff, TiffContext *context, GError **error)
{
        guchar *pixels = NULL;
        gint width, height, rowstride, bytes;
        ...
        rowstride = width * 4;
        if (rowstride / 4 != width) { /* overflow */
                g_set_error_literal (error,
                                     GDK_PIXBUF_ERROR,
                                     GDK_PIXBUF_ERROR_CORRUPT_IMAGE,
                                     _("Dimensions of TIFF image too large"));
                return NULL;
        }

        bytes = height * rowstride;
        if (bytes / rowstride != height) { /* overflow */
                g_set_error_literal (error,
                                     GDK_PIXBUF_ERROR,
                                     GDK_PIXBUF_ERROR_CORRUPT_IMAGE,
                                     _("Dimensions of TIFF image too large"));
                return NULL;
        }


Both these overflow checks are eliminated by the compiler. I do not have a patch for fixing these two overflow checks.