GNOME Bugzilla – Bug 305894
excessive memory use in gdk_pixbuf_new_from_file_XXX functions
Last modified: 2010-07-10 04:08:33 UTC
Three test programs which load 2005_15.jpg: 111712708 bytes, 11477x7965, 274242915 bytes unpacked image size. you may get this file at http://imgsrc.hubblesite.org/hu/db/2005/12/images/a/formats/full_jpg.jpg First program just loads image from current directory, other two programs resize it (shrink to 1600x1200) while loading. Surprisingly, shrinking uses 280 *more* memory. ================================================= # cat t_file.c ================================================= #include <stdio.h> #include <gdk-pixbuf/gdk-pixbuf.h> #define filename "2005_15.jpg" int main() { GdkPixbuf* pb; g_type_init(); pb = gdk_pixbuf_new_from_file(filename, NULL); printf("Pixbuf created, pb=%p\n", pb); getchar(); gdk_pixbuf_unref(pb); printf("Pixbuf freed\n", pb); getchar(); return 0; } ================================================= # cat t_size.c ================================================= #include <stdio.h> #include <gdk-pixbuf/gdk-pixbuf.h> #define filename "2005_15.jpg" int main() { GdkPixbuf* pb; g_type_init(); pb = gdk_pixbuf_new_from_file_at_size(filename, 1600, 1200, NULL); printf("Pixbuf created, pb=%p\n", pb); getchar(); gdk_pixbuf_unref(pb); printf("Pixbuf freed\n", pb); getchar(); return 0; } ================================================= # cat t_scale.c ================================================= #include <stdio.h> #include <gdk-pixbuf/gdk-pixbuf.h> #define filename "2005_15.jpg" int main() { GdkPixbuf* pb; g_type_init(); pb = gdk_pixbuf_new_from_file_at_scale(filename, 1600, 1200, 0, NULL); printf("Pixbuf created, pb=%p\n", pb); getchar(); gdk_pixbuf_unref(pb); printf("Pixbuf freed\n", pb); getchar(); return 0; } ================================================= # time ./t_file </dev/null; time ./t_size </dev/null; time ./t_scale </dev/null Pixbuf created, pb=0x80a8700 Pixbuf freed real 0m15.782s user 0m14.316s sys 0m0.843s Pixbuf created, pb=0x804ea08 Pixbuf freed real 0m9.762s user 0m8.062s sys 0m1.346s Pixbuf created, pb=0x804ea08 Pixbuf freed real 0m9.746s user 0m8.008s sys 0m1.392s Memory usage during above test (note: it's *total* memory usage for all processes combined, mostly X+KDE): 21:32:17.9 mem 43M 21:32:18.0 mem 43M 21:32:18.1 mem 43M 21:32:18.2 mem 43M 21:32:18.3 mem 43M 21:32:18.4 mem 43M 21:32:18.5 mem 43M 21:32:18.6 mem 43M 21:32:18.7 mem 43M 21:32:18.8 mem 43M 21:32:18.9 mem 44M 21:32:19.0 mem 46M 21:32:19.1 mem 48M 21:32:19.2 mem 49M 21:32:19.3 mem 51M 21:32:19.4 mem 53M 21:32:19.5 mem 54M 21:32:19.6 mem 56M 21:32:19.7 mem 58M 21:32:19.8 mem 59M 21:32:19.9 mem 61M 21:32:20.0 mem 62M 21:32:20.1 mem 64M 21:32:20.2 mem 66M 21:32:20.3 mem 67M 21:32:20.4 mem 69M 21:32:20.5 mem 71M 21:32:20.6 mem 72M 21:32:20.7 mem 74M 21:32:20.8 mem 75M 21:32:20.9 mem 77M 21:32:21.0 mem 79M 21:32:21.1 mem 80M 21:32:21.2 mem 82M 21:32:21.3 mem 83M 21:32:21.4 mem 85M 21:32:21.5 mem 87M 21:32:21.6 mem 88M 21:32:21.7 mem 90M 21:32:21.8 mem 92M 21:32:21.9 mem 93M 21:32:22.0 mem 95M 21:32:22.1 mem 97M 21:32:22.2 mem 98M 21:32:22.3 mem 100M 21:32:22.4 mem 102M 21:32:22.5 mem 103M 21:32:22.6 mem 105M 21:32:22.7 mem 107M 21:32:22.8 mem 108M 21:32:22.9 mem 110M 21:32:23.0 mem 112M 21:32:23.1 mem 113M 21:32:23.2 mem 115M 21:32:23.3 mem 117M 21:32:23.4 mem 118M 21:32:23.5 mem 120M 21:32:23.6 mem 122M 21:32:23.7 mem 123M 21:32:23.8 mem 125M 21:32:23.9 mem 127M 21:32:24.0 mem 128M 21:32:24.1 mem 130M 21:32:24.2 mem 132M 21:32:24.3 mem 134M 21:32:24.4 mem 135M 21:32:24.5 mem 137M 21:32:24.6 mem 139M 21:32:24.7 mem 141M 21:32:24.8 mem 142M 21:32:24.9 mem 144M 21:32:25.0 mem 145M 21:32:25.1 mem 147M 21:32:25.2 mem 149M 21:32:25.3 mem 150M 21:32:25.4 mem 152M 21:32:25.5 mem 154M 21:32:25.6 mem 156M 21:32:25.7 mem 157M 21:32:25.8 mem 159M 21:32:25.9 mem 161M 21:32:26.0 mem 162M 21:32:26.1 mem 164M 21:32:26.2 mem 166M 21:32:26.3 mem 167M 21:32:26.4 mem 169M 21:32:26.5 mem 171M 21:32:26.6 mem 172M 21:32:26.7 mem 174M 21:32:26.8 mem 176M 21:32:26.9 mem 178M 21:32:27.0 mem 179M 21:32:27.1 mem 181M 21:32:27.2 mem 183M 21:32:27.3 mem 185M 21:32:27.4 mem 186M 21:32:27.5 mem 188M 21:32:27.6 mem 190M 21:32:27.7 mem 191M 21:32:27.8 mem 193M 21:32:27.9 mem 195M 21:32:28.0 mem 196M 21:32:28.1 mem 198M 21:32:28.2 mem 200M 21:32:28.3 mem 201M 21:32:28.4 mem 203M 21:32:28.5 mem 205M 21:32:28.6 mem 206M 21:32:28.7 mem 208M 21:32:28.8 mem 210M 21:32:28.9 mem 211M 21:32:29.0 mem 213M 21:32:29.1 mem 215M 21:32:29.2 mem 217M 21:32:29.3 mem 218M 21:32:29.4 mem 220M 21:32:29.5 mem 221M 21:32:29.6 mem 223M 21:32:29.7 mem 225M 21:32:29.8 mem 227M 21:32:29.9 mem 228M 21:32:30.0 mem 229M 21:32:30.1 mem 231M 21:32:30.2 mem 233M 21:32:30.3 mem 235M 21:32:30.4 mem 236M 21:32:30.5 mem 238M 21:32:30.6 mem 239M 21:32:30.7 mem 241M 21:32:30.8 mem 243M 21:32:30.9 mem 244M 21:32:31.0 mem 246M 21:32:31.1 mem 248M 21:32:31.2 mem 249M 21:32:31.3 mem 251M 21:32:31.4 mem 253M 21:32:31.5 mem 254M 21:32:31.6 mem 256M 21:32:31.7 mem 258M 21:32:31.8 mem 259M 21:32:31.9 mem 261M 21:32:32.0 mem 263M 21:32:32.1 mem 264M 21:32:32.2 mem 266M 21:32:32.3 mem 268M 21:32:32.4 mem 269M 21:32:32.5 mem 271M 21:32:32.6 mem 272M 21:32:32.7 mem 274M 21:32:32.8 mem 276M 21:32:32.9 mem 277M 21:32:33.0 mem 279M 21:32:33.1 mem 281M 21:32:33.2 mem 282M 21:32:33.3 mem 284M 21:32:33.4 mem 286M 21:32:33.5 mem 287M 21:32:33.6 mem 289M 21:32:33.7 mem 291M 21:32:33.8 mem 292M 21:32:33.9 mem 294M 21:32:34.0 mem 296M 21:32:34.1 mem 298M 21:32:34.2 mem 299M 21:32:34.3 mem 301M 21:32:34.4 mem 303M 21:32:34.5 mem 304M 21:32:34.6 mem 306M t_file run completed. ~300Mb was used. 21:32:34.7 mem 46M 21:32:34.8 mem 51M 21:32:34.9 mem 57M 21:32:35.0 mem 62M 21:32:35.1 mem 68M 21:32:35.2 mem 74M 21:32:35.3 mem 80M 21:32:35.4 mem 86M 21:32:35.5 mem 92M 21:32:35.6 mem 97M 21:32:35.7 mem 103M 21:32:35.8 mem 109M 21:32:35.9 mem 115M 21:32:36.0 mem 121M 21:32:36.1 mem 126M 21:32:36.2 mem 132M 21:32:36.3 mem 137M 21:32:36.4 mem 143M 21:32:36.5 mem 149M 21:32:36.6 mem 154M 21:32:36.7 mem 160M 21:32:36.8 mem 166M 21:32:36.9 mem 172M 21:32:37.0 mem 178M 21:32:37.1 mem 184M 21:32:37.2 mem 189M 21:32:37.3 mem 195M 21:32:37.4 mem 201M 21:32:37.5 mem 207M 21:32:37.6 mem 213M 21:32:37.7 mem 219M 21:32:37.8 mem 225M 21:32:37.9 mem 232M 21:32:38.0 mem 237M 21:32:38.1 mem 243M 21:32:38.2 mem 249M 21:32:38.3 mem 255M 21:32:38.4 mem 261M 21:32:38.5 mem 267M 21:32:38.6 mem 273M 21:32:38.7 mem 279M 21:32:38.8 mem 285M 21:32:38.9 mem 291M 21:32:39.0 mem 297M 21:32:39.1 mem 304M 21:32:39.2 mem 310M 21:32:39.3 mem 316M 21:32:39.4 mem 322M 21:32:39.5 mem 328M 21:32:39.6 mem 334M 21:32:39.7 mem 340M 21:32:39.8 mem 346M 21:32:39.9 mem 352M 21:32:40.0 mem 357M 21:32:40.1 mem 363M 21:32:40.2 mem 368M 21:32:40.3 mem 374M 21:32:40.4 mem 380M 21:32:40.5 mem 386M 21:32:40.6 mem 391M 21:32:40.7 mem 397M 21:32:40.8 mem 403M 21:32:40.9 mem 409M 21:32:41.0 mem 415M 21:32:41.1 mem 421M 21:32:41.2 mem 426M 21:32:41.3 mem 432M 21:32:41.4 mem 437M 21:32:41.5 mem 443M 21:32:41.6 mem 448M 21:32:41.7 mem 454M 21:32:41.8 mem 460M 21:32:41.9 mem 465M 21:32:42.0 mem 471M 21:32:42.1 mem 477M 21:32:42.2 mem 483M 21:32:42.3 mem 489M 21:32:42.4 mem 494M 21:32:42.5 mem 500M 21:32:42.6 mem 505M 21:32:42.7 mem 511M 21:32:42.8 mem 517M 21:32:42.9 mem 523M 21:32:43.0 mem 529M 21:32:43.1 mem 535M 21:32:43.2 mem 541M 21:32:43.3 mem 546M 21:32:43.4 mem 552M 21:32:43.5 mem 558M 21:32:43.6 mem 563M 21:32:43.7 mem 569M 21:32:43.8 mem 575M 21:32:43.9 mem 581M Why t_size needs 280M more memory than t_file?! 21:32:44.0 mem 282M 21:32:44.1 mem 61M 21:32:44.2 mem 63M 21:32:44.3 mem 64M 21:32:44.4 mem 65M 21:32:44.5 mem 48M 21:32:44.6 mem 54M 21:32:44.7 mem 59M 21:32:44.8 mem 65M 21:32:44.9 mem 71M 21:32:45.0 mem 76M 21:32:45.1 mem 82M 21:32:45.2 mem 88M 21:32:45.3 mem 94M 21:32:45.4 mem 100M 21:32:45.5 mem 106M 21:32:45.6 mem 111M 21:32:45.7 mem 117M 21:32:45.8 mem 123M 21:32:45.9 mem 129M 21:32:46.0 mem 135M 21:32:46.1 mem 141M 21:32:46.2 mem 146M 21:32:46.3 mem 152M 21:32:46.4 mem 158M 21:32:46.5 mem 163M 21:32:46.6 mem 169M 21:32:46.7 mem 175M 21:32:46.8 mem 181M 21:32:46.9 mem 186M 21:32:47.0 mem 192M 21:32:47.1 mem 198M 21:32:47.2 mem 204M 21:32:47.3 mem 210M 21:32:47.4 mem 216M 21:32:47.5 mem 222M 21:32:47.6 mem 228M 21:32:47.7 mem 235M 21:32:47.8 mem 241M 21:32:47.9 mem 247M 21:32:48.0 mem 253M 21:32:48.1 mem 259M 21:32:48.2 mem 265M 21:32:48.3 mem 271M 21:32:48.4 mem 277M 21:32:48.5 mem 282M 21:32:48.6 mem 288M 21:32:48.7 mem 295M 21:32:48.8 mem 301M 21:32:48.9 mem 307M 21:32:49.0 mem 313M 21:32:49.1 mem 320M 21:32:49.2 mem 326M 21:32:49.3 mem 332M 21:32:49.4 mem 338M 21:32:49.5 mem 344M 21:32:49.6 mem 350M 21:32:49.7 mem 356M 21:32:49.8 mem 362M 21:32:49.9 mem 368M 21:32:50.0 mem 373M 21:32:50.1 mem 378M 21:32:50.2 mem 384M 21:32:50.3 mem 390M 21:32:50.4 mem 396M 21:32:50.5 mem 401M 21:32:50.6 mem 407M 21:32:50.7 mem 413M 21:32:50.8 mem 419M 21:32:50.9 mem 425M 21:32:51.0 mem 431M 21:32:51.1 mem 437M 21:32:51.2 mem 442M 21:32:51.3 mem 447M 21:32:51.4 mem 453M 21:32:51.5 mem 458M 21:32:51.6 mem 464M 21:32:51.7 mem 470M 21:32:51.8 mem 476M 21:32:51.9 mem 482M 21:32:52.0 mem 487M 21:32:52.1 mem 493M 21:32:52.2 mem 499M 21:32:52.3 mem 504M 21:32:52.4 mem 510M 21:32:52.5 mem 516M 21:32:52.6 mem 522M 21:32:52.7 mem 528M 21:32:52.8 mem 533M 21:32:52.9 mem 539M 21:32:53.0 mem 545M 21:32:53.1 mem 551M 21:32:53.2 mem 557M 21:32:53.3 mem 563M 21:32:53.4 mem 569M 21:32:53.5 mem 574M 21:32:53.6 mem 580M 21:32:53.7 mem 437M Same for t_scale 21:32:53.8 mem 61M 21:32:53.9 mem 62M 21:32:54.0 mem 64M 21:32:54.1 mem 65M 21:32:54.2 mem 43M 21:32:54.3 mem 43M 21:32:54.4 mem 43M 21:32:54.5 mem 43M
It seems that gdk-puxbuf jpeg loader wants progressive jpegs to look neat by drawing then as they are loaded. Thus "buffered_image mode" (in libjpeg-speak) is used which allows for that. That "buffered mode" is of no gain for single-pass jpegs, but for loader simplicity sake it is used for them also. The fact that libjpeg uses up to 2 times uncompressed image size (!) in this mode until image is fully loaded wasn't noticed (read libjpeg.doc in lib source). This is very bad if you are loading multi megabyte single-pass jpeg. libjpeg eats 2 x image_size and gtk eats another 1 x image_size (seen with gqview, a gtk-based image viewer).
Created attachment 47058 [details] [review] do not use buffered mode This patch was lightly run-tested with gqview. ~3x mem usage reduction. Full-fledged solution may use buffered mode for progressive jpegs and "normal" mode for single-pass ones. Further improvements (todo): libjpeg can downsample images as they are loaded, this will allow loading of arbitrarily large files (even those which are much larger than total virtual memory)
Yes, I think we should use buffered mode for multi-pass images. An additional refinement would be to only use buffered mode if we are actually loading incrementally. The downsampling is actually used for the load_at_size functionality.
Comfirming because of last comments. Raising priority because of patch. We are looking forward to a refined patch :)
Created attachment 52860 [details] [review] Add code to handle non-buffered mode This patch add a different branch to handle non-buffered mode loading because certain functions shouldn't be called then. The other part is to disable the buffered mode unconditionally, but it should be made user-definable. With this patch downscaling normal jpegs does not consume any significant amounts of extra memory, the target pixbuf is the dominant allocation. For progressive jpegs libjpeg will still allocate one buffer that is three times the resolution of the source image.
Created attachment 55941 [details] [review] Updated patch from maemo-gtk New patch that actually looks at cinfo->progressive_mode instead of always setting cinfo->buffered_mode to FALSE.
a test image for progressive kpeg loading should be added to gtk. then i don't see further problems with applying the patch.
Fixed in CVS: 2005-12-14 Michael Natterer <mitch@imendio.com> * gdk-pixbuf/io-jpeg.c: applied patch from maemo-gtk which avoids the allocation of an intermediate buffer for non-progressive jpegs. Fixed bug #305894. * tests/test-images/valid_jpeg_progressive_test: new test image so we can test both loading code paths in io-jpeg.c