GNOME Bugzilla – Bug 725177
Memory isn't freed after downscaling
Last modified: 2018-05-24 14:19:28 UTC
To make a test for this bug report (https://bugzilla.gnome.org/show_bug.cgi?id=719619) I have scaled up an image with the dimension of 1680x1050 to 53760x33600. This process was successfull and my system ended up in using all of his 16 GiB of memory with 4 GiB of memory in the swap. To make another test I have scaled down this image back to 1680x1050 but after the downscaling finished GIMP has still used this huge amount of memory. This is especially a problem because GIMP claimed on exporting it hasn't enough memory to export the image.
The memory is on the undo stack. Try clearing it via "Windows -> Dockable -> Undo History". Is the memory freed now?
Here is the test: - Opening an image with the dimension of 1680x1050 -> GIMP uses ~83 MiB. - Scaling it up to 2400% on both sites without interpolation -> GIMP uses ~9.4 GiB. - Scaling it down to the original size with Sinc (Lanczos3) -> GIMP uses ~10.2 GiB. - On trying to clear all journal entries GIMP tells me that this operation would free 4.1 GB of memory and on confirming this action -> GIMP uses ~9.7 GiB. Not all memory is freed but even if it would I have thought about another problem: As I said in the startpost GIMP has claimed that there is not enough memory to export the image but this message could be wrong. The reason is: - On my system overcommiting is enabled so it is unlikely that malloc or another of these functions returned NULL because of an OOM. - This makes me thinking of that GIMP measures the memory usage on a higher level (maybe the Linux Kernel API or /proc ?). - In this case GIMP hasn't considered that I had enough swap available that the exporting would cause GIMP to swap something out to successfully finsish the exporting.
Can you be more specific on that warning on export? Which file format did you export to? I'm not aware of such warning code.
After the system memory is nearly empty (but there is still ~10 GiB free swap available) this message appears on exporting as PNG: Unable to run plug-in "file-png" (/usr/lib/gimp/2.0/plug-ins/file-png) Failed to fork (Cannot allocate memory)
That is very strange because a fork() copies almost nothing of the parent process, most of the memory should be copy-on-write. In the same situation, can you do a memory-consuming GIMP operation, like creating a big new image?
I have tested this again and even after the same error on exporting appeared I could successfully create an image with the dimension of 20000x20000 pixel.
Does this still happen with 2.8.14?
Yes, exporting an image does still fail and the memory on cleaning the history is also still not fully freed.
I have figured out that the memory in use comes from glibc's caching methods. I have created a ticket for this: https://sourceware.org/bugzilla/show_bug.cgi?id=18504
We do call mallopt (M_MMAP_THRESHOLD, TILE_WIDTH * TILE_HEIGHT); and IIRC this should make allocations larger than the threshold being returned to the system. Smaller stuff will still be cached. At least that's how it used to work years ago when we added that code.
Setting the environment variable MALLOC_MMAP_MAX_ to a very high value causes that GIMP uses on my system after doing the reproduction steps only ~200 MiB instead of ~9 GiB. So without setting this environment variable at least the most parts of the ~9 GiB seems to be on the data segment and that it doesn't get released.
So it seems that things still can be properly configured malloc-wise, but that our way of doing it has stopped working... Since you are standing knee deep in malloc docs, could you perhaps figure an API to call to make GIMP actually release memory chunks larger than a certain threshold? Maybe that call should be done in GEGL generically, where the tile size is configured now. But in this bug we are still talking about 2.8.14, right?
(In reply to Michael Natterer from comment #12) > So it seems that things still can be properly configured malloc-wise, > but that our way of doing it has stopped working... glibc can be controlled with some environment variables but a call of mallopt() should overwrite these. And just in case you have misread it: I have pointed to another variable which you have pointed to so if GIMP doesn't also set M_MMAP_MAX nothing was unexpected here. (In reply to Michael Natterer from comment #12) > Since you are standing knee deep in malloc docs, could you perhaps > figure an API to call to make GIMP actually release memory chunks > larger than a certain threshold? malloc_trim() does this and takes your requested threshold as an argument (it should only release free memory at the top of the heap but see the next passage below). (In reply to Michael Natterer from comment #12) > Maybe that call should be done in GEGL generically, where the tile > size is configured now. For now I wouldn't go that far as based on the glibc ticket there may be something weird. Normally free() should call this function conditionally but in my testcase it doesn't. Likely because the top of the heap was not free but in this case malloc_trim() should also not free anything but it did on my tests. I'm assuming either there is something broken in glibc or the documentation isn't fully correct. Maybe we should wait until the glibc ticket is handled and focus here on the issue that the exporting fails on high memory usage. Otherwise maybe you want to check if you can reproduce this issue on Linux (I'm on Linux 3.19.8 (x86_64) with glibc 2.21) and if malloc_trim() could be used as a workaround. (In reply to Michael Natterer from comment #12) > But in this bug we are still talking about 2.8.14, right? Yes.
(In reply to Michael Natterer from comment #10) > We do call > > mallopt (M_MMAP_THRESHOLD, TILE_WIDTH * TILE_HEIGHT); > > and IIRC this should make allocations larger than the threshold being > returned to the system. Smaller stuff will still be cached. At least > that's how it used to work years ago when we added that code. According to http://man7.org/linux/man-pages/man3/mallopt.3.html : > M_MMAP_THRESHOLD > : > : > Note: Nowadays, glibc uses a dynamic mmap threshold by > default. The initial value of the threshold is 128*1024, but > when blocks larger than the current threshold and less than or > equal to DEFAULT_MMAP_THRESHOLD_MAX are freed, the threshold > is adjusted upward to the size of the freed block. When > dynamic mmap thresholding is in effect, the threshold for > trimming the heap is also dynamically adjusted to be twice the > dynamic mmap threshold. Dynamic adjustment of the mmap > threshold is disabled if any of the M_TRIM_THRESHOLD, > M_TOP_PAD, M_MMAP_THRESHOLD, or M_MMAP_MAX parameters is set. M_MMAP_THRESHOLD has an initial value of 128*1024, but, in GIMP, the *dynamic* adjustment will probably keep raising the threshold to the largest GeglBuffer so far freed. (Which is not helpful!) As the last line says, dynamic adjustment is disabled if any of those options is set, so I tried starting GIMP thus: $ MALLOC_MMAP_THRESHOLD_=128*1024 gimp-2.9 --verbose and sure enough a great deal of memory (though not all) is freed upon closing the image and also quite a lot is freed upon resetting undo history. But it is still NOT being freed upon completion of each GEGL op.
*** Bug 760020 has been marked as a duplicate of this bug. ***
I found this bug report while looking for a solution or a workaround to unreleased memory with Gimp 2.9.5, used on Ubuntu 14.04 I use to open and close many Gimp documents during a working session. Rapidly the memory fills up to levels like 10 GiB (on a 16 GiB system), and refreshing of images begins to crawl. Releasing the undo stack or closing documents does nothing. At the end I can have an empty Gimp window with a 10 GiB footprint. Fortunately MALLOC_MMAP_THRESHOLD_=128*1024 does the job for me. Memory is "reasonably" released, I am not feeling anymore like hitting a usability wall. But I was not able to reproduce this behavior with Gimp 2.8.10 (Ubuntu 14.04) or 2.8.14 (Ubuntu 15.04). Memory is released when closing documents or deleting the undo stack. So I wonder if it should not be reported as a standalone bug for 2.9.5?
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/gimp/issues/538.