After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 725177 - Memory isn't freed after downscaling
Memory isn't freed after downscaling
Status: RESOLVED OBSOLETE
Product: GIMP
Classification: Other
Component: General
2.8.14
Other Linux
: Normal normal
: ---
Assigned To: GIMP Bugs
GIMP Bugs
: 760020 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2014-02-25 22:54 UTC by sworddragon2
Modified: 2018-05-24 14:19 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description sworddragon2 2014-02-25 22:54:31 UTC
To make a test for this bug report (https://bugzilla.gnome.org/show_bug.cgi?id=719619) I have scaled up an image with the dimension of 1680x1050 to 53760x33600. This process was successfull and my system ended up in using all of his 16 GiB of memory with 4 GiB of memory in the swap. To make another test I have scaled down this image back to 1680x1050 but after the downscaling finished GIMP has still used this huge amount of memory. This is especially a problem because GIMP claimed on exporting it hasn't enough memory to export the image.
Comment 1 Michael Natterer 2014-02-26 00:35:24 UTC
The memory is on the undo stack. Try clearing it via
"Windows -> Dockable -> Undo History". Is the memory freed now?
Comment 2 sworddragon2 2014-02-26 23:32:17 UTC
Here is the test:

- Opening an image with the dimension of 1680x1050 -> GIMP uses ~83 MiB.
- Scaling it up to 2400% on both sites without interpolation -> GIMP uses ~9.4 GiB.
- Scaling it down to the original size with Sinc (Lanczos3) -> GIMP uses ~10.2 GiB.
- On trying to clear all journal entries GIMP tells me that this operation would free 4.1 GB of memory and on confirming this action -> GIMP uses ~9.7 GiB.


Not all memory is freed but even if it would I have thought about another problem: As I said in the startpost GIMP has claimed that there is not enough memory to export the image but this message could be wrong. The reason is:

- On my system overcommiting is enabled so it is unlikely that malloc or another of these functions returned NULL because of an OOM.
- This makes me thinking of that GIMP measures the memory usage on a higher level (maybe the Linux Kernel API or /proc ?).
- In this case GIMP hasn't considered that I had enough swap available that the exporting would cause GIMP to swap something out to successfully finsish the exporting.
Comment 3 Michael Natterer 2014-04-08 23:33:09 UTC
Can you be more specific on that warning on export? Which file format
did you export to? I'm not aware of such warning code.
Comment 4 sworddragon2 2014-04-09 23:12:01 UTC
After the system memory is nearly empty (but there is still ~10 GiB free swap available) this message appears on exporting as PNG:

Unable to run plug-in "file-png"
(/usr/lib/gimp/2.0/plug-ins/file-png)

Failed to fork (Cannot allocate memory)
Comment 5 Michael Natterer 2014-04-10 12:05:43 UTC
That is very strange because a fork() copies almost nothing of the
parent process, most of the memory should be copy-on-write.

In the same situation, can you do a memory-consuming GIMP operation,
like creating a big new image?
Comment 6 sworddragon2 2014-04-11 16:57:12 UTC
I have tested this again and even after the same error on exporting appeared I could successfully create an image with the dimension of 20000x20000 pixel.
Comment 7 Michael Schumacher 2015-06-06 17:16:16 UTC
Does this still happen with 2.8.14?
Comment 8 sworddragon2 2015-06-06 23:05:49 UTC
Yes, exporting an image does still fail and the memory on cleaning the history is also still not fully freed.
Comment 9 sworddragon2 2015-06-09 01:03:34 UTC
I have figured out that the memory in use comes from glibc's caching methods. I have created a ticket for this: https://sourceware.org/bugzilla/show_bug.cgi?id=18504
Comment 10 Michael Natterer 2015-06-09 06:17:33 UTC
We do call

mallopt (M_MMAP_THRESHOLD, TILE_WIDTH * TILE_HEIGHT);

and IIRC this should make allocations larger than the threshold being
returned to the system. Smaller stuff will still be cached. At least
that's how it used to work years ago when we added that code.
Comment 11 sworddragon2 2015-06-09 09:20:50 UTC
Setting the environment variable MALLOC_MMAP_MAX_ to a very high value causes that GIMP uses on my system after doing the reproduction steps only ~200 MiB instead of ~9 GiB. So without setting this environment variable at least the most parts of the ~9 GiB seems to be on the data segment and that it doesn't get released.
Comment 12 Michael Natterer 2015-06-09 09:56:08 UTC
So it seems that things still can be properly configured malloc-wise,
but that our way of doing it has stopped working...

Since you are standing knee deep in malloc docs, could you perhaps
figure an API to call to make GIMP actually release memory chunks
larger than a certain threshold?

Maybe that call should be done in GEGL generically, where the tile
size is configured now.

But in this bug we are still talking about 2.8.14, right?
Comment 13 sworddragon2 2015-06-09 10:39:45 UTC
(In reply to Michael Natterer from comment #12)
> So it seems that things still can be properly configured malloc-wise,
> but that our way of doing it has stopped working...

glibc can be controlled with some environment variables but a call of mallopt() should overwrite these. And just in case you have misread it: I have pointed to another variable which you have pointed to so if GIMP doesn't also set M_MMAP_MAX nothing was unexpected here.


(In reply to Michael Natterer from comment #12)
> Since you are standing knee deep in malloc docs, could you perhaps
> figure an API to call to make GIMP actually release memory chunks
> larger than a certain threshold?

malloc_trim() does this and takes your requested threshold as an argument (it should only release free memory at the top of the heap but see the next passage below).


(In reply to Michael Natterer from comment #12)
> Maybe that call should be done in GEGL generically, where the tile
> size is configured now.

For now I wouldn't go that far as based on the glibc ticket there may be something weird. Normally free() should call this function conditionally but in my testcase it doesn't. Likely because the top of the heap was not free but in this case malloc_trim() should also not free anything but it did on my tests. I'm assuming either there is something broken in glibc or the documentation isn't fully correct. Maybe we should wait until the glibc ticket is handled and focus here on the issue that the exporting fails on high memory usage. Otherwise maybe you want to check if you can reproduce this issue on Linux (I'm on Linux 3.19.8 (x86_64) with glibc 2.21) and if malloc_trim() could be used as a workaround.


(In reply to Michael Natterer from comment #12)
> But in this bug we are still talking about 2.8.14, right?

Yes.
Comment 14 Jonathan Tait 2016-01-01 11:56:40 UTC
(In reply to Michael Natterer from comment #10)
> We do call
> 
> mallopt (M_MMAP_THRESHOLD, TILE_WIDTH * TILE_HEIGHT);
> 
> and IIRC this should make allocations larger than the threshold being
> returned to the system. Smaller stuff will still be cached. At least
> that's how it used to work years ago when we added that code.

According to http://man7.org/linux/man-pages/man3/mallopt.3.html :

> M_MMAP_THRESHOLD
>  :
>  :
> Note: Nowadays, glibc uses a dynamic mmap threshold by
> default.  The initial value of the threshold is 128*1024, but
> when blocks larger than the current threshold and less than or
> equal to DEFAULT_MMAP_THRESHOLD_MAX are freed, the threshold
> is adjusted upward to the size of the freed block.  When
> dynamic mmap thresholding is in effect, the threshold for
> trimming the heap is also dynamically adjusted to be twice the
> dynamic mmap threshold.  Dynamic adjustment of the mmap
> threshold is disabled if any of the M_TRIM_THRESHOLD,
> M_TOP_PAD, M_MMAP_THRESHOLD, or M_MMAP_MAX parameters is set.

M_MMAP_THRESHOLD has an initial value of 128*1024, but, in GIMP, the *dynamic* adjustment will probably keep raising the threshold to the largest GeglBuffer so far freed. (Which is not helpful!)

As the last line says, dynamic adjustment is disabled if any of those options is set, so I tried starting GIMP thus:

$ MALLOC_MMAP_THRESHOLD_=128*1024 gimp-2.9 --verbose

and sure enough a great deal of memory (though not all) is freed upon closing the image and also quite a lot is freed upon resetting undo history. But it is still NOT being freed upon completion of each GEGL op.
Comment 15 Michael Natterer 2016-01-04 18:02:19 UTC
*** Bug 760020 has been marked as a duplicate of this bug. ***
Comment 16 Patrice St-Gelais 2017-02-07 04:40:10 UTC
I found this bug report while looking for a solution or a workaround to unreleased memory with Gimp 2.9.5, used on Ubuntu 14.04
I use to open and close many Gimp documents during a working session. Rapidly the memory fills up to levels like 10 GiB (on a 16 GiB system), and refreshing of images begins to crawl. Releasing the undo stack or closing documents does nothing. At the end I can have an empty Gimp window with a 10 GiB footprint.

Fortunately MALLOC_MMAP_THRESHOLD_=128*1024 does the job for me. Memory is "reasonably" released, I am not feeling anymore like hitting a usability wall.

But I was not able to reproduce this behavior with Gimp 2.8.10 (Ubuntu 14.04) or 2.8.14 (Ubuntu 15.04). Memory is released when closing documents or deleting the undo stack.

So I wonder if it should not be reported as a standalone bug for 2.9.5?
Comment 17 GNOME Infrastructure Team 2018-05-24 14:19:28 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/gimp/issues/538.