GIMP has a dashboard dockable (Windows -> Dockable Dialogs -> Dashboard) which can be used to monitor GEGL swap usage.  When swap space is running low (< 10% left), GIMP raises the dashboard to alert about that.  Note that you have to keep the dashboard open for this to happen (its tab doesn't have to be visible, it just needs to be open in one of the docks.)

Note that the swap file is per-session, i.e., it doesn't accumulate across sessions, and it should be deleted when you close GIMP; leftover swap files (which might not have gotten deleted on exit because GIMP crashed, etc.) should be deleted on startup as well.  A 140 GiB swap file is pretty unusual, but not impossible.  Working on very large images, or having many images open, can lead to high swap usage.  It's also possible that this is the result of a bug: e.g., some rogue operation that started eating up all the space.  It'd be interesting to know what you were doing at the time the space ran out.

Dealing with an out-of-swap-space condition is tricky: once swap space completely runs out, you start losing data (to the point that saving the image at this point is a bad idea, since some of the saved data might be garbage -- definitely don't save over the existing file.)  Therefore, even if there was a way to limit the swap size, there's the question of what would be a reasonable action to take once the swap hits the limit, other than to increase the limit?  This is why a low-space warning is probably more useful than a hard limit.

That being said, a way to limit the swap size is something we should probably have anyway, if only to prevent GEGL from eating up all free disk space when something goes wrong.

The location of the swap file can be controlled though the GEGL_SWAP env var, but a GIMP configuration option is probably a good idea.  Note that you can effectively limit the swap size by putting it in a dedicated file-system with limited size: either a separate partition, a loop device, etc.

(In reply to Ell from comment #1)
> GIMP has a dashboard dockable (Windows -> Dockable Dialogs -> Dashboard)
> which can be used to monitor GEGL swap usage.  When swap space is running
> low (< 10% left), GIMP raises the dashboard to alert about that.  Note that
> you have to keep the dashboard open for this to happen (its tab doesn't have
> to be visible, it just needs to be open in one of the docks.)

Ah, thanks! I didn't know about the dashboard.

> Note that the swap file is per-session, i.e., it doesn't accumulate across
> sessions, and it should be deleted when you close GIMP; leftover swap files
> (which might not have gotten deleted on exit because GIMP crashed, etc.)
> should be deleted on startup as well.  A 140 GiB swap file is pretty
> unusual, but not impossible.  Working on very large images, or having many
> images open, can lead to high swap usage.  It's also possible that this is
> the result of a bug: e.g., some rogue operation that started eating up all
> the space.  It'd be interesting to know what you were doing at the time the
> space ran out.

I only had one image open. At the start of the session I had opened another image by mistake, but closed it immediately and opened the one I wanted. The size on disk of each of these two images is under 600MB, though each image does have quite a few layers, with alpha channels for all the layers and masks for many of the layers (currently 16 layers each with an alpha channel, 4 groups, 3 layer masks, and 6 selections saved as channels). It's a painting, so I really am using all of the alpha channels.

When GEGL created the 140GiB swap file, I was using the paintbrush somewhat, and the "Warp Transform" tool a lot, to move pixels around, making whole lot of small moves to push stripes on a cloth into the proper folds. I've never done this type of painting before, so it was a slow process that took up a lot of time. So I did use the Warp Transform tool quite a lot. Maybe this is what caused such a huge swap file?

> Dealing with an out-of-swap-space condition is tricky: once swap space
> completely runs out, you start losing data (to the point that saving the
> image at this point is a bad idea, since some of the saved data might be
> garbage -- definitely don't save over the existing file.)  Therefore, even
> if there was a way to limit the swap size, there's the question of what
> would be a reasonable action to take once the swap hits the limit, other
> than to increase the limit?  This is why a low-space warning is probably
> more useful than a hard limit.
> 

This sounds reasonable. Could GEGL reuse swap space, overwrite previously written stuff on a first in/first out basis? Or maybe rotate sequentially through several different swap files, each with its own fixed maximum size, overwriting what's in each previously-used swap file? Or does GEGL somehow need every bit of information that it ever writes to the swap drive during a given editing session?

> That being said, a way to limit the swap size is something we should
> probably have anyway, if only to prevent GEGL from eating up all free disk
> space when something goes wrong.
> 
> The location of the swap file can be controlled though the GEGL_SWAP env
> var, but a GIMP configuration option is probably a good idea.  Note that you
> can effectively limit the swap size by putting it in a dedicated file-system
> with limited size: either a separate partition, a loop device, etc.

But it sounds like putting the swap file on a limited-size partition (which every partition is) won't keep GEGL from crashing if it runs out of room. I have some spare 200GB external drives, but writing through a USB connection would be very slow.

Currently (still using the Warp Transform tool):

 * GIMP is using 20GB out of 32GB RAM according to htop. 
 * The undo history is almost empty and never reported using more than 1.1GB RAM.
 * The dashboard says the cache is at 6.4GB with a limit (set in Preferences) of 16GB. 
 * The dashboard says the swap is at 0GB. 
 * GIMP reports on the status bar that the image is occupying 9.1GB RAM.

So I wonder how GEGL/GIMP are using the "20-6.4=13.6" additional GB RAM reported by htop? That's 20GB "user" reported by htop, with free 5806 MB, cached 5362 MB, and buffers 106440 kB. 

Anyway, I wasn't paying attention to my taskbar RAM monitor when GEGL ran out of swap. But when it did run out of swap I noticed the taskbar RAM monitor looked like it was at 100%, something I've never seen before.

(In reply to Elle Stone from comment #2)
> (In reply to Ell from comment #1)
> > Note that the swap file is per-session, i.e., it doesn't accumulate across
> > sessions, and it should be deleted when you close GIMP; leftover swap files
> > (which might not have gotten deleted on exit because GIMP crashed, etc.)
> > should be deleted on startup as well.  A 140 GiB swap file is pretty
> > unusual, but not impossible.  Working on very large images, or having many
> > images open, can lead to high swap usage.  It's also possible that this is
> > the result of a bug: e.g., some rogue operation that started eating up all
> > the space.  It'd be interesting to know what you were doing at the time the
> > space ran out.
> 
> I only had one image open. At the start of the session I had opened another
> image by mistake, but closed it immediately and opened the one I wanted. The
> size on disk of each of these two images is under 600MB, though each image
> does have quite a few layers, with alpha channels for all the layers and
> masks for many of the layers (currently 16 layers each with an alpha
> channel, 4 groups, 3 layer masks, and 6 selections saved as channels). It's
> a painting, so I really am using all of the alpha channels.
> 
> When GEGL created the 140GiB swap file, I was using the paintbrush somewhat,
> and the "Warp Transform" tool a lot, to move pixels around, making whole lot
> of small moves to push stripes on a cloth into the proper folds. I've never
> done this type of painting before, so it was a slow process that took up a
> lot of time. So I did use the Warp Transform tool quite a lot. Maybe this is
> what caused such a huge swap file?

None of this sounds like it should result in a swap file that big.  The warp tool is quite a memory guzzler, though.  IIRC it keeps an unlimited undo history, so each stroke you make eats up a chunk of memory.  All this memory should be freed, however, once you commit, so it would still take using the warp tool for a very long time, without committing in the middle, to reach this much memory.  If you do commit often, it doesn't really matter how much you use it.  We should probably put an upper limit on the warp tool's undo stack regardless.

> > Dealing with an out-of-swap-space condition is tricky: once swap space
> > completely runs out, you start losing data (to the point that saving the
> > image at this point is a bad idea, since some of the saved data might be
> > garbage -- definitely don't save over the existing file.)  Therefore, even
> > if there was a way to limit the swap size, there's the question of what
> > would be a reasonable action to take once the swap hits the limit, other
> > than to increase the limit?  This is why a low-space warning is probably
> > more useful than a hard limit.
> > 
> 
> This sounds reasonable. Could GEGL reuse swap space, overwrite previously
> written stuff on a first in/first out basis? Or maybe rotate sequentially
> through several different swap files, each with its own fixed maximum size,
> overwriting what's in each previously-used swap file? Or does GEGL somehow
> need every bit of information that it ever writes to the swap drive during a
> given editing session?

GEGL does reuse both cache and swap space.  When data is no longer needed (e.g., when you close an image, clear the undo history, etc.) it's discarded, and the space it occupied is later reused.  However, the size of the swap file on disk never shrinks during a session.  Note that the dashboard has both an "occupied" field and a "size" field in the swap section.  The latter denotes the size of the swap file on disk, and the former denotes how much of that GEGL actually uses.  When the swap file fills up, the size of the file increases; when data is released, GEGL remembers that the corresponding portions of the swap file can be reused (hence the "occupied" field decreases), but the file itself remains the same size (hence the "size" field doesn't change).

> > The location of the swap file can be controlled though the GEGL_SWAP env
> > var, but a GIMP configuration option is probably a good idea.  Note that you
> > can effectively limit the swap size by putting it in a dedicated file-system
> > with limited size: either a separate partition, a loop device, etc.
> 
> But it sounds like putting the swap file on a limited-size partition (which
> every partition is) won't keep GEGL from crashing if it runs out of room. I
> have some spare 200GB external drives, but writing through a USB connection
> would be very slow.

Sure, the swap file hitting the limit will always be a problem, but if you wanted to limit it regardless, that's one option.  With as much memory as you have, I wouldn't worry too much about the speed of accessing the swap, but I would go for a loop device (basically, a file system that lives inside a regular file in your normal file system), rather than an external USB drive.

> Currently (still using the Warp Transform tool):
> 
>  * GIMP is using 20GB out of 32GB RAM according to htop. 
>  * The undo history is almost empty and never reported using more than 1.1GB
> RAM.
>  * The dashboard says the cache is at 6.4GB with a limit (set in
> Preferences) of 16GB. 
>  * The dashboard says the swap is at 0GB. 
>  * GIMP reports on the status bar that the image is occupying 9.1GB RAM.

The size reported on the status bar and in the undo history dialog is a very rough, liberal estimate (in the sense that it tends to overshoot), and can be quite wrong.  In contrast, the numbers reported in the dashboard are accurate, but they only measure the amount of memory used by GEGL for storing tiles -- actual pixel data.  It doesn't include metadata (the data needed to manage the data), and it doesn't include any other memory that GIMP might be using for other things.  Still, the tile data should represent the bulk of the memory GIMP is using.

> So I wonder how GEGL/GIMP are using the "20-6.4=13.6" additional GB RAM
> reported by htop? That's 20GB "user" reported by htop, with free 5806 MB,
> cached 5362 MB, and buffers 106440 kB. 

Memory is handled by different layers, each of which has its own view of what it considers "free", and what it considers "in use".  top shows how much memory the OS considers GIMP to be using, while the dashboard shows how much memory GEGL actually makes use of (for storing tiles).  Explaining discrepancies between the different numbers is hard without really looking into it.

> Anyway, I wasn't paying attention to my taskbar RAM monitor when GEGL ran
> out of swap. But when it did run out of swap I noticed the taskbar RAM
> monitor looked like it was at 100%, something I've never seen before.

This might indicate that we have a bug somewhere.  When the cache fills up, GEGL starts moving data to the swap, but it doesn't wait for the data to be fully written to disk.  While the data is pending writing to disk, it's kept in-memory, but isn't accounted for when calculating the cache usage.  Normally, the data is written to disk fast enough, freeing its memory, so it's not a problem, but if some "cancerous" operation starts producing unlimited amounts of data, at a rate higher than in which it can be written to disk, then the RAM, as well as the disk, might fill up.

-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/gegl/issues/63.