After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 625609 - housekeeping: Needs to be a bit more clever
housekeeping: Needs to be a bit more clever
Status: RESOLVED FIXED
Product: gnome-settings-daemon
Classification: Core
Component: housekeeping
2.30.x
Other Linux
: Normal normal
: ---
Assigned To: gnome-settings-daemon-maint
gnome-settings-daemon-maint
3.10
Depends on:
Blocks:
 
 
Reported: 2010-07-29 19:06 UTC by MarkG
Modified: 2013-09-30 15:39 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
iotop screenshot (13.70 KB, image/png)
2011-07-04 00:00 UTC, André V
  Details
patch to properly allow disabling of thumbnail cache cleaning (1.85 KB, patch)
2013-09-30 14:26 UTC, Mark Ellis
accepted-commit_now Details | Review
housekeeping: Optimise for "do nothing" when cleaning thumbnails (1.92 KB, patch)
2013-09-30 15:31 UTC, Bastien Nocera
committed Details | Review

Description MarkG 2010-07-29 19:06:55 UTC
The original bug report for this is at:

https://bugs.launchpad.net/ubuntu/+source/gnome-settings-daemon/+bug/597512

The Ubuntu guys asked me to post it here too; I couldn't see any similar bugs in the current list.

When I log into my laptop I regularly see huge CPU usage of which only a small part is doing anything useful and 75% is spent in I/O wait. The culprit is the gnome-settings-daemon, and the whole system is unusable until it stops what it's doing.

What it appears to be doing according to strace is calling lstat() on thousands and thousands and thousands of stored thumbnail images, as fast as it can do so. This prevents anything else from accessing the disk.

I presume you're trying to find the oldest files and delete them, which makes sense. But I believe there are a number of problems here:

1. Calling lstat() on thousands and thousands of files without a break is going to completely trash system performance unless you have an SSD, at least on ext3; newer filesystems may be more efficient at directory reads but I don't think you can rely on that. This is particularly problematic when it's run when the user logs in and is eager to use the computer to do something as they have to wait thirty seconds or more before it's usable.

2. I suspect this performance hit is made even worse when using ecryptfs, since that's adding an extra layer or three between lstat() and the actual hard disk.

3. I'm sure I read somewhere that Ubuntu set the thumbnail lifetime higher than other Linux distros, so it's going to have even more files to scan.

4. There's also the question of whether you really want to be putting 20,000+ thumbnail files into a single directory in the first place? Putting them into multiple directories and scanning individual directories on an intermittent basis would seem more sensible.

This doesn't happen every time I log in, but when it does happen it's painful.
Comment 1 Bastien Nocera 2010-10-12 18:10:37 UTC
That's the code in the housekeeping plugin that purges old files, and trims the size of the thumbnail cache. See plugins/housekeeping/gsd-housekeeping-manager.c:purge_thumbnail_cache().

The code to cleanup on shutdown is also utterly stupid, as it will list, and sort the files, before removing every single one of them.

This code could really do with moving to GIO...

About 4), that still wouldn't fix the problem of listing and removing those files...
Comment 2 karlrt 2010-12-14 11:25:18 UTC
Downstreambug on launchpad: https://bugs.launchpad.net/gnome-settings-daemon/+bug/505085
Comment 3 Hagen Fürstenau 2011-06-16 07:51:30 UTC
At the very least, purge_thumbnail_cache() could check whether "max_age" and "max_size" are both < 0 BEFORE actually reading the thumnail dirs! If it did, you could at least disable thumbnail purging by setting both to -1 with gconf-editor. 

Presently, the only way to avoid a performance hit (e.g. with a large thumbnail cache under ecryptfs) seems to be disabling the whole housekeeping plugin.
Comment 4 André V 2011-07-04 00:00:53 UTC
Created attachment 191205 [details]
iotop screenshot
Comment 5 André V 2011-07-04 00:03:30 UTC
How can I disable the plugin while this isn't solved?

gnome-settings-daemon makes my system almost unresponsive for about 8 minutes on every startup…

$ find .thumbnails | wc
  30486   30486 1736812
$ du -sm .thumbnails
819	.thumbnails
Comment 6 Hagen Fürstenau 2011-07-04 07:40:51 UTC
workaround: start gconf-editor and untick

/apps/gnome_settings_daemon/plugins/housekeeping/active

Apparently this will also disable the low disk space check and warning (and maybe something else as well?), but I guess this isn't essential...
Comment 7 André V 2011-07-04 13:49:15 UTC
Thanks! For me it was in dconf, /org/gnome/settings-daemon/plugins/housekeeping/

I wasn't finding it in gconf, so I checked the files the package installed, found the org.gnome.settings-daemon.plugins.housekeeping.gschema.xml and inside there's the path="/org/gnome/settings-daemon/plugins/housekeeping/". :)

I should have said I'm using the version 3.1.2-0ubuntu1 of gnome-settings-daemon, Ubuntu oneiric (development branch).
Comment 8 ricflomag 2011-08-14 23:35:17 UTC
Noob suggestion: why not fix this bug just by runnig the house keeping thread with an appropriate ionice priority ?
Comment 9 Kamil Páral 2011-11-24 21:21:33 UTC
Still problem in GNOME 3.2. Someone with sufficient privileges (like the reporter), please change Version to 3.2 or to master. Thanks.
Comment 10 Bastien Nocera 2011-11-24 22:22:28 UTC
(In reply to comment #9)
> Still problem in GNOME 3.2. Someone with sufficient privileges (like the
> reporter), please change Version to 3.2 or to master. Thanks.

There's no need to do that. If the bug wasn't current anymore, it would be closed.
Comment 11 Mark Ellis 2013-09-30 14:26:09 UTC
Created attachment 256088 [details] [review]
patch to properly allow disabling of thumbnail cache cleaning

Per comment #3, the attached patch checks if max_age and max_size are both -1, ie thumbnail cleaning is disabled, before scanning the thumbnail dirs. This enables the performance hit to be removed through a simple configuration option.
Comment 12 Bastien Nocera 2013-09-30 14:35:32 UTC
Review of attachment 256088 [details] [review]:

Looks good otherwise for gnome-3-8 and master.

::: gnome-settings-daemon.orig/plugins/housekeeping/gsd-housekeeping-manager.c
@@ +230,3 @@
+
+        /* if both are set to -1, we don't need to read anything */
+        if ((purge_data.max_age < 0) && (purge_data.max_size < 0))

No need for the extra brackets.
Comment 13 Bastien Nocera 2013-09-30 15:31:06 UTC
Created attachment 256099 [details] [review]
housekeeping: Optimise for "do nothing" when cleaning thumbnails

If max_age and max_size are both -1, thumbnail cleaning
is disabled, so there is no need to scan the cache
directories.

https://bugzilla.gnome.org/show_bug.cgi?id=625609
https://bugs.launchpad.net/ubuntu/+source/gnome-settings-daemon/+bug/505085
Comment 14 Bastien Nocera 2013-09-30 15:33:05 UTC
Review of attachment 256088 [details] [review]:

> The referenced bugs concern a serious
> performance issue at login, which is caused by the
> thumbnail cache scan.

You might be looking in the wrong place for this, the purge only runs 2 minutes after g-s-d's startup. I've removed this from the commit message.

Furthermore, please attach patches in git format, not whatever format Ubuntu packages use, please.
Comment 15 Bastien Nocera 2013-09-30 15:39:41 UTC
Attachment 256099 [details] pushed as 12f2121 - housekeeping: Optimise for "do nothing" when cleaning thumbnails