After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 305894 - excessive memory use in gdk_pixbuf_new_from_file_XXX functions
excessive memory use in gdk_pixbuf_new_from_file_XXX functions
Status: RESOLVED FIXED
Product: gdk-pixbuf
Classification: Platform
Component: general
git master
Other All
: High normal
: ---
Assigned To: gtk-bugs
gtk-bugs
Depends on:
Blocks: 169867
 
 
Reported: 2005-05-30 05:37 UTC by Denis Vlasenko
Modified: 2010-07-10 04:08 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
do not use buffered mode (4.19 KB, patch)
2005-05-31 05:54 UTC, Denis Vlasenko
none Details | Review
Add code to handle non-buffered mode (2.08 KB, patch)
2005-09-30 16:01 UTC, Tommi Komulainen
none Details | Review
Updated patch from maemo-gtk (2.84 KB, patch)
2005-12-13 13:40 UTC, Michael Natterer
committed Details | Review

Description Denis Vlasenko 2005-05-30 05:37:00 UTC
Three test programs which load 2005_15.jpg:
111712708 bytes, 11477x7965, 274242915 bytes unpacked image size.
you may get this file at
http://imgsrc.hubblesite.org/hu/db/2005/12/images/a/formats/full_jpg.jpg

First program just loads image from current directory,
other two programs resize it (shrink to 1600x1200) while loading.

Surprisingly, shrinking uses 280 *more* memory.

=================================================
# cat t_file.c
=================================================
#include <stdio.h>
#include <gdk-pixbuf/gdk-pixbuf.h>

#define filename "2005_15.jpg"

int main() {
    GdkPixbuf* pb;

    g_type_init();

    pb = gdk_pixbuf_new_from_file(filename, NULL);
    printf("Pixbuf created, pb=%p\n", pb);
    getchar();

    gdk_pixbuf_unref(pb);
    printf("Pixbuf freed\n", pb);
    getchar();

    return 0;
}

=================================================
# cat t_size.c
=================================================
#include <stdio.h>
#include <gdk-pixbuf/gdk-pixbuf.h>

#define filename "2005_15.jpg"

int main() {
    GdkPixbuf* pb;

    g_type_init();

    pb = gdk_pixbuf_new_from_file_at_size(filename, 1600, 1200, NULL);
    printf("Pixbuf created, pb=%p\n", pb);
    getchar();

    gdk_pixbuf_unref(pb);
    printf("Pixbuf freed\n", pb);
    getchar();

    return 0;
}

=================================================
# cat t_scale.c
=================================================
#include <stdio.h>
#include <gdk-pixbuf/gdk-pixbuf.h>

#define filename "2005_15.jpg"

int main() {
    GdkPixbuf* pb;

    g_type_init();

    pb = gdk_pixbuf_new_from_file_at_scale(filename, 1600, 1200, 0, NULL);
    printf("Pixbuf created, pb=%p\n", pb);
    getchar();

    gdk_pixbuf_unref(pb);
    printf("Pixbuf freed\n", pb);
    getchar();

    return 0;
}

=================================================
# time ./t_file </dev/null; time ./t_size </dev/null; time ./t_scale </dev/null
Pixbuf created, pb=0x80a8700
Pixbuf freed

real    0m15.782s
user    0m14.316s
sys     0m0.843s
Pixbuf created, pb=0x804ea08
Pixbuf freed

real    0m9.762s
user    0m8.062s
sys     0m1.346s
Pixbuf created, pb=0x804ea08
Pixbuf freed

real    0m9.746s
user    0m8.008s
sys     0m1.392s

Memory usage during above test
(note: it's *total* memory usage for all processes
combined, mostly X+KDE):

21:32:17.9 mem  43M
21:32:18.0 mem  43M
21:32:18.1 mem  43M
21:32:18.2 mem  43M
21:32:18.3 mem  43M
21:32:18.4 mem  43M
21:32:18.5 mem  43M
21:32:18.6 mem  43M
21:32:18.7 mem  43M
21:32:18.8 mem  43M
21:32:18.9 mem  44M
21:32:19.0 mem  46M
21:32:19.1 mem  48M
21:32:19.2 mem  49M
21:32:19.3 mem  51M
21:32:19.4 mem  53M
21:32:19.5 mem  54M
21:32:19.6 mem  56M
21:32:19.7 mem  58M
21:32:19.8 mem  59M
21:32:19.9 mem  61M
21:32:20.0 mem  62M
21:32:20.1 mem  64M
21:32:20.2 mem  66M
21:32:20.3 mem  67M
21:32:20.4 mem  69M
21:32:20.5 mem  71M
21:32:20.6 mem  72M
21:32:20.7 mem  74M
21:32:20.8 mem  75M
21:32:20.9 mem  77M
21:32:21.0 mem  79M
21:32:21.1 mem  80M
21:32:21.2 mem  82M
21:32:21.3 mem  83M
21:32:21.4 mem  85M
21:32:21.5 mem  87M
21:32:21.6 mem  88M
21:32:21.7 mem  90M
21:32:21.8 mem  92M
21:32:21.9 mem  93M
21:32:22.0 mem  95M
21:32:22.1 mem  97M
21:32:22.2 mem  98M
21:32:22.3 mem 100M
21:32:22.4 mem 102M
21:32:22.5 mem 103M
21:32:22.6 mem 105M
21:32:22.7 mem 107M
21:32:22.8 mem 108M
21:32:22.9 mem 110M
21:32:23.0 mem 112M
21:32:23.1 mem 113M
21:32:23.2 mem 115M
21:32:23.3 mem 117M
21:32:23.4 mem 118M
21:32:23.5 mem 120M
21:32:23.6 mem 122M
21:32:23.7 mem 123M
21:32:23.8 mem 125M
21:32:23.9 mem 127M
21:32:24.0 mem 128M
21:32:24.1 mem 130M
21:32:24.2 mem 132M
21:32:24.3 mem 134M
21:32:24.4 mem 135M
21:32:24.5 mem 137M
21:32:24.6 mem 139M
21:32:24.7 mem 141M
21:32:24.8 mem 142M
21:32:24.9 mem 144M
21:32:25.0 mem 145M
21:32:25.1 mem 147M
21:32:25.2 mem 149M
21:32:25.3 mem 150M
21:32:25.4 mem 152M
21:32:25.5 mem 154M
21:32:25.6 mem 156M
21:32:25.7 mem 157M
21:32:25.8 mem 159M
21:32:25.9 mem 161M
21:32:26.0 mem 162M
21:32:26.1 mem 164M
21:32:26.2 mem 166M
21:32:26.3 mem 167M
21:32:26.4 mem 169M
21:32:26.5 mem 171M
21:32:26.6 mem 172M
21:32:26.7 mem 174M
21:32:26.8 mem 176M
21:32:26.9 mem 178M
21:32:27.0 mem 179M
21:32:27.1 mem 181M
21:32:27.2 mem 183M
21:32:27.3 mem 185M
21:32:27.4 mem 186M
21:32:27.5 mem 188M
21:32:27.6 mem 190M
21:32:27.7 mem 191M
21:32:27.8 mem 193M
21:32:27.9 mem 195M
21:32:28.0 mem 196M
21:32:28.1 mem 198M
21:32:28.2 mem 200M
21:32:28.3 mem 201M
21:32:28.4 mem 203M
21:32:28.5 mem 205M
21:32:28.6 mem 206M
21:32:28.7 mem 208M
21:32:28.8 mem 210M
21:32:28.9 mem 211M
21:32:29.0 mem 213M
21:32:29.1 mem 215M
21:32:29.2 mem 217M
21:32:29.3 mem 218M
21:32:29.4 mem 220M
21:32:29.5 mem 221M
21:32:29.6 mem 223M
21:32:29.7 mem 225M
21:32:29.8 mem 227M
21:32:29.9 mem 228M
21:32:30.0 mem 229M
21:32:30.1 mem 231M
21:32:30.2 mem 233M
21:32:30.3 mem 235M
21:32:30.4 mem 236M
21:32:30.5 mem 238M
21:32:30.6 mem 239M
21:32:30.7 mem 241M
21:32:30.8 mem 243M
21:32:30.9 mem 244M
21:32:31.0 mem 246M
21:32:31.1 mem 248M
21:32:31.2 mem 249M
21:32:31.3 mem 251M
21:32:31.4 mem 253M
21:32:31.5 mem 254M
21:32:31.6 mem 256M
21:32:31.7 mem 258M
21:32:31.8 mem 259M
21:32:31.9 mem 261M
21:32:32.0 mem 263M
21:32:32.1 mem 264M
21:32:32.2 mem 266M
21:32:32.3 mem 268M
21:32:32.4 mem 269M
21:32:32.5 mem 271M
21:32:32.6 mem 272M
21:32:32.7 mem 274M
21:32:32.8 mem 276M
21:32:32.9 mem 277M
21:32:33.0 mem 279M
21:32:33.1 mem 281M
21:32:33.2 mem 282M
21:32:33.3 mem 284M
21:32:33.4 mem 286M
21:32:33.5 mem 287M
21:32:33.6 mem 289M
21:32:33.7 mem 291M
21:32:33.8 mem 292M
21:32:33.9 mem 294M
21:32:34.0 mem 296M
21:32:34.1 mem 298M
21:32:34.2 mem 299M
21:32:34.3 mem 301M
21:32:34.4 mem 303M
21:32:34.5 mem 304M
21:32:34.6 mem 306M

t_file run completed. ~300Mb was used.

21:32:34.7 mem  46M
21:32:34.8 mem  51M
21:32:34.9 mem  57M
21:32:35.0 mem  62M
21:32:35.1 mem  68M
21:32:35.2 mem  74M
21:32:35.3 mem  80M
21:32:35.4 mem  86M
21:32:35.5 mem  92M
21:32:35.6 mem  97M
21:32:35.7 mem 103M
21:32:35.8 mem 109M
21:32:35.9 mem 115M
21:32:36.0 mem 121M
21:32:36.1 mem 126M
21:32:36.2 mem 132M
21:32:36.3 mem 137M
21:32:36.4 mem 143M
21:32:36.5 mem 149M
21:32:36.6 mem 154M
21:32:36.7 mem 160M
21:32:36.8 mem 166M
21:32:36.9 mem 172M
21:32:37.0 mem 178M
21:32:37.1 mem 184M
21:32:37.2 mem 189M
21:32:37.3 mem 195M
21:32:37.4 mem 201M
21:32:37.5 mem 207M
21:32:37.6 mem 213M
21:32:37.7 mem 219M
21:32:37.8 mem 225M
21:32:37.9 mem 232M
21:32:38.0 mem 237M
21:32:38.1 mem 243M
21:32:38.2 mem 249M
21:32:38.3 mem 255M
21:32:38.4 mem 261M
21:32:38.5 mem 267M
21:32:38.6 mem 273M
21:32:38.7 mem 279M
21:32:38.8 mem 285M
21:32:38.9 mem 291M
21:32:39.0 mem 297M
21:32:39.1 mem 304M
21:32:39.2 mem 310M
21:32:39.3 mem 316M
21:32:39.4 mem 322M
21:32:39.5 mem 328M
21:32:39.6 mem 334M
21:32:39.7 mem 340M
21:32:39.8 mem 346M
21:32:39.9 mem 352M
21:32:40.0 mem 357M
21:32:40.1 mem 363M
21:32:40.2 mem 368M
21:32:40.3 mem 374M
21:32:40.4 mem 380M
21:32:40.5 mem 386M
21:32:40.6 mem 391M
21:32:40.7 mem 397M
21:32:40.8 mem 403M
21:32:40.9 mem 409M
21:32:41.0 mem 415M
21:32:41.1 mem 421M
21:32:41.2 mem 426M
21:32:41.3 mem 432M
21:32:41.4 mem 437M
21:32:41.5 mem 443M
21:32:41.6 mem 448M
21:32:41.7 mem 454M
21:32:41.8 mem 460M
21:32:41.9 mem 465M
21:32:42.0 mem 471M
21:32:42.1 mem 477M
21:32:42.2 mem 483M
21:32:42.3 mem 489M
21:32:42.4 mem 494M
21:32:42.5 mem 500M
21:32:42.6 mem 505M
21:32:42.7 mem 511M
21:32:42.8 mem 517M
21:32:42.9 mem 523M
21:32:43.0 mem 529M
21:32:43.1 mem 535M
21:32:43.2 mem 541M
21:32:43.3 mem 546M
21:32:43.4 mem 552M
21:32:43.5 mem 558M
21:32:43.6 mem 563M
21:32:43.7 mem 569M
21:32:43.8 mem 575M
21:32:43.9 mem 581M

Why t_size needs 280M more memory than t_file?!

21:32:44.0 mem 282M
21:32:44.1 mem  61M
21:32:44.2 mem  63M
21:32:44.3 mem  64M
21:32:44.4 mem  65M
21:32:44.5 mem  48M
21:32:44.6 mem  54M
21:32:44.7 mem  59M
21:32:44.8 mem  65M
21:32:44.9 mem  71M
21:32:45.0 mem  76M
21:32:45.1 mem  82M
21:32:45.2 mem  88M
21:32:45.3 mem  94M
21:32:45.4 mem 100M
21:32:45.5 mem 106M
21:32:45.6 mem 111M
21:32:45.7 mem 117M
21:32:45.8 mem 123M
21:32:45.9 mem 129M
21:32:46.0 mem 135M
21:32:46.1 mem 141M
21:32:46.2 mem 146M
21:32:46.3 mem 152M
21:32:46.4 mem 158M
21:32:46.5 mem 163M
21:32:46.6 mem 169M
21:32:46.7 mem 175M
21:32:46.8 mem 181M
21:32:46.9 mem 186M
21:32:47.0 mem 192M
21:32:47.1 mem 198M
21:32:47.2 mem 204M
21:32:47.3 mem 210M
21:32:47.4 mem 216M
21:32:47.5 mem 222M
21:32:47.6 mem 228M
21:32:47.7 mem 235M
21:32:47.8 mem 241M
21:32:47.9 mem 247M
21:32:48.0 mem 253M
21:32:48.1 mem 259M
21:32:48.2 mem 265M
21:32:48.3 mem 271M
21:32:48.4 mem 277M
21:32:48.5 mem 282M
21:32:48.6 mem 288M
21:32:48.7 mem 295M
21:32:48.8 mem 301M
21:32:48.9 mem 307M
21:32:49.0 mem 313M
21:32:49.1 mem 320M
21:32:49.2 mem 326M
21:32:49.3 mem 332M
21:32:49.4 mem 338M
21:32:49.5 mem 344M
21:32:49.6 mem 350M
21:32:49.7 mem 356M
21:32:49.8 mem 362M
21:32:49.9 mem 368M
21:32:50.0 mem 373M
21:32:50.1 mem 378M
21:32:50.2 mem 384M
21:32:50.3 mem 390M
21:32:50.4 mem 396M
21:32:50.5 mem 401M
21:32:50.6 mem 407M
21:32:50.7 mem 413M
21:32:50.8 mem 419M
21:32:50.9 mem 425M
21:32:51.0 mem 431M
21:32:51.1 mem 437M
21:32:51.2 mem 442M
21:32:51.3 mem 447M
21:32:51.4 mem 453M
21:32:51.5 mem 458M
21:32:51.6 mem 464M
21:32:51.7 mem 470M
21:32:51.8 mem 476M
21:32:51.9 mem 482M
21:32:52.0 mem 487M
21:32:52.1 mem 493M
21:32:52.2 mem 499M
21:32:52.3 mem 504M
21:32:52.4 mem 510M
21:32:52.5 mem 516M
21:32:52.6 mem 522M
21:32:52.7 mem 528M
21:32:52.8 mem 533M
21:32:52.9 mem 539M
21:32:53.0 mem 545M
21:32:53.1 mem 551M
21:32:53.2 mem 557M
21:32:53.3 mem 563M
21:32:53.4 mem 569M
21:32:53.5 mem 574M
21:32:53.6 mem 580M
21:32:53.7 mem 437M

Same for t_scale

21:32:53.8 mem  61M
21:32:53.9 mem  62M
21:32:54.0 mem  64M
21:32:54.1 mem  65M
21:32:54.2 mem  43M
21:32:54.3 mem  43M
21:32:54.4 mem  43M
21:32:54.5 mem  43M
Comment 1 Denis Vlasenko 2005-05-31 05:47:53 UTC
It seems that gdk-puxbuf jpeg loader wants progressive jpegs to look neat
by drawing then as they are loaded. Thus "buffered_image mode"
(in libjpeg-speak) is used which allows for that.

That "buffered mode" is of no gain for single-pass jpegs, but for
loader simplicity sake it is used for them also.

The fact that libjpeg uses up to 2 times uncompressed image size (!)
in this mode until image is fully loaded wasn't noticed
(read libjpeg.doc in lib source).

This is very bad if you are loading multi megabyte single-pass jpeg.
libjpeg eats 2 x image_size and gtk eats another 1 x image_size
(seen with gqview, a gtk-based image viewer).
Comment 2 Denis Vlasenko 2005-05-31 05:54:26 UTC
Created attachment 47058 [details] [review]
do not use buffered mode

This patch was lightly run-tested with gqview. ~3x mem usage reduction.
Full-fledged solution may use buffered mode for progressive jpegs
and "normal" mode for single-pass ones.

Further improvements (todo): libjpeg can downsample images as they are
loaded, this will allow loading of arbitrarily large files (even
those which are much larger than total virtual memory)
Comment 3 Matthias Clasen 2005-06-07 17:11:23 UTC
Yes, I think we should use buffered mode for multi-pass images. An additional
refinement would be to only use buffered mode if we are actually loading
incrementally. The downsampling is actually used for the load_at_size
functionality.
Comment 4 Christian Kirbach 2005-06-09 12:30:45 UTC
Comfirming because of last comments.

Raising priority because of patch.

We are looking forward to a refined patch :)
Comment 5 Tommi Komulainen 2005-09-30 16:01:37 UTC
Created attachment 52860 [details] [review]
Add code to handle non-buffered mode

This patch add a different branch to handle non-buffered mode loading because
certain functions shouldn't be called then. The other part is to disable the
buffered mode unconditionally, but it should be made user-definable.

With this patch downscaling normal jpegs does not consume any significant
amounts of extra memory, the target pixbuf is the dominant allocation.

For progressive jpegs libjpeg will still allocate one buffer that is three
times the resolution of the source image.
Comment 6 Michael Natterer 2005-12-13 13:40:06 UTC
Created attachment 55941 [details] [review]
Updated patch from maemo-gtk

New patch that actually looks at cinfo->progressive_mode instead
of always setting cinfo->buffered_mode to FALSE.
Comment 7 Tim Janik 2005-12-14 12:01:13 UTC
a test image for progressive kpeg loading should be added to gtk. then i don't
see further problems with applying the patch.
Comment 8 Michael Natterer 2005-12-14 12:48:35 UTC
Fixed in CVS:

2005-12-14  Michael Natterer  <mitch@imendio.com>

	* gdk-pixbuf/io-jpeg.c: applied patch from maemo-gtk which avoids
	the allocation of an intermediate buffer for non-progressive
	jpegs. Fixed bug #305894.

	* tests/test-images/valid_jpeg_progressive_test: new test image so
	we can test both loading code paths in io-jpeg.c