After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 766135 - Memory leak in gnome-settings-daemom ubuntu 16.04
Memory leak in gnome-settings-daemom ubuntu 16.04
Status: RESOLVED OBSOLETE
Product: gnome-settings-daemon
Classification: Core
Component: general
3.18.x
Other Linux
: Normal normal
: ---
Assigned To: gnome-settings-daemon-maint
gnome-settings-daemon-maint
Depends on:
Blocks:
 
 
Reported: 2016-05-08 12:25 UTC by Damien GERARD
Modified: 2019-03-20 11:35 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
htop - processes (507.17 KB, image/jpeg)
2016-05-08 12:25 UTC, Damien GERARD
Details
Screenshot of memleak (152.38 KB, image/png)
2016-10-01 14:06 UTC, Michael Sandler
Details
xrandr-mich (1.72 KB, text/plain)
2016-10-01 14:07 UTC, Michael Sandler
Details
debug logs (270.79 KB, application/x-7z-compressed)
2016-10-07 16:24 UTC, Michael Sandler
Details

Description Damien GERARD 2016-05-08 12:25:27 UTC
Created attachment 327469 [details]
htop - processes

Hi :)

After installinng kubuntu, I switched to gnome-desktop (don't know if relevant and if can help).
However after some time, all gnome animations become slower and slower. I did not use my laptop more than 4 hours in a row so I don't know if related to all suspend, but anyway as you can see in the screenshot, there is a nice mem leak.

distrib: ubuntu 16.04

Waiting for a fix, any solution to restart it without having to log off / login again ?

Thanks !
Comment 1 Bastien Nocera 2016-05-08 15:53:55 UTC
Which exact version of gnome-settings-daemon is that?
Comment 2 Damien GERARD 2016-05-09 04:37:15 UTC
3.18.2 according `apt-get cache`.

And from the package sources as well:
https://launchpad.net/ubuntu/xenial/+source/gnome-settings-daemon
Comment 3 Bastien Nocera 2016-05-09 11:42:36 UTC
You should test with 3.18.3. If you see the same problem with that version, try disabling the plugins one-by-one with dconf-editor, under /org/gnome/settings-daemon/plugins in the UI.

The "active" setting is the one you want to disable. You'd disable one, reboot, test, and if it still shows the problem, re-enable it, and reboot again.

Or you can run gnome-settings-daemon under valgrind, but that's more involved.
Comment 4 Bastien Nocera 2016-05-09 11:46:15 UTC
Oh, and you can start with disabling the "updates" plugin, which is an Ubuntu only change.
Comment 5 Damien GERARD 2016-05-10 13:57:56 UTC
I am now monitoring this process. Nothing to do with the suspend state. It just grows up little by little with a normal use (firefox/thunderbird/terminal). I will try with a more recent version of 3.18.x as soon as I find the corresponding package.
Comment 6 Damien GERARD 2016-05-16 15:32:35 UTC
My system slows down actually for sure when it tries to turnoff my external monitor

May 17 00:20:21 hatchery /usr/lib/gdm3/gdm-x-session[1483]: (II) intel(0): switch to mode 3840x2160@30.0 on HDMI1 using pipe 0, position (1920, 0), rotation normal, reflection none
May 17 00:20:21 hatchery /usr/lib/gdm3/gdm-x-session[1483]: (II) intel(0): switch to mode 1920x1080@60.0 on eDP1 using pipe 1, position (0, 0), rotation normal, reflection none
May 17 00:20:22 hatchery /usr/lib/gdm3/gdm-x-session[1483]: (--) intel(0): HDMI max TMDS frequency 300000KHz
May 17 00:20:49 hatchery /usr/lib/gdm3/gdm-x-session[1483]: (II) intel(0): resizing framebuffer to 1920x1080
May 17 00:20:49 hatchery /usr/lib/gdm3/gdm-x-session[1483]: (--) intel(0): HDMI max TMDS frequency 300000KHz
May 17 00:20:49 hatchery /usr/lib/gdm3/gdm-x-session[1483]: (--) intel(0): HDMI max TMDS frequency 300000KHz
May 17 00:20:49 hatchery /usr/lib/gdm3/gdm-x-session[1483]: (II) intel(0): resizing framebuffer to 5760x2160

(http://pastebin.com/SHnJgt1T)
Comment 7 Damien GERARD 2016-05-16 15:34:00 UTC
the memory usage usage of gnome-settings-daemon is directly linked to those events
Comment 8 Michael Sandler 2016-10-01 14:06:51 UTC
Created attachment 336728 [details]
Screenshot of memleak
Comment 9 Michael Sandler 2016-10-01 14:07:13 UTC
Created attachment 336729 [details]
xrandr-mich
Comment 10 Michael Sandler 2016-10-01 14:07:52 UTC
I have the same memory leak problem on Arch Linux.
Here are some facts I've noticed about it:

* The leak only happens when I have external monitors connected to the laptop
* The leak does appear to start when I leave the laptop and the monitors shut down after inactivity, as already pointed out by Damien GERARD
* The only way I found to overcome this, is to logoff/login back, which is not a really good solution, I've tried using the --replace switch, but it only restarts the gnome-settings-daemon under my user - although it leaks too, the one that really swallows all of my memory is the on running under the gdm user

Some info on my machine:
$ pacman -Q gnome-settings-daemon 
gnome-settings-daemon 3.20.1+8+g40bf4fd-1

xrandr output is attached, explanation of monitors:
HDMI1 ==> Physical HDMI port on my laptop, screen number 2
HDMI2 ==> Mini DisplayPort on laptop ==> HDMI adapter, screen number 3
eDP1 ==> Internal display (Turned off when I use external monitors)

Screenshot of leak is attached.

Please, if I need to supply more data, or help debug this, tell me what to do, I'd really like to help solve this issue, it is very annoying.
Comment 11 Damien GERARD 2016-10-01 14:11:17 UTC
It was an external monitor in my case as well, connected in hdmi.
Comment 12 Michael Sandler 2016-10-02 12:58:03 UTC
I decided to dig a bit deeper into the problem, and now when the daemon got to around 2.7GB in memory, I used gdb to make a coredump

$ pgrep -f gnome-settings-daemon -u gdm
942
# gdb -p 942
(gdb) generate-core-file
Saved corefile core.2071
(gdb) quit
$ ls -alh core.942
-rw-r--r-- 1 root root 2.7G Oct  2 13:11 core.942

Then I got all the strings from it, and took out the ones that appear a lot of times

$ strings core.942 | sort | uniq -c | sed -r 's/\s+([0-9]+)\s(.+)/\1,"\2"/' | sort -nr -t ',' -k 1,2 | head -n 42
601012,"uaua{sv})"
173958,"7`qV"
171232,"underscanning"
171028,"primary"
171027,"serial"
171027,"backlight"
171026,"width-mm"
171026,"vendor"
171026,"supports-underscanning"
171026,"product"
171026,"presentation"
171026,"min-backlight-step"
171026,"height-mm"
171026,"display-name"
171026,"connector-type"
168774,"edid"
125272,"RD24L04140564"
116072,"RD24L"
115499,"auaua{sv})"
113038,"HDMIA"
110869,"      "
110227,"q8-@X,E"
107324,"DJNC4JA000171"
105642,"i2352Vh"
61452,"DBus"
58477,"4p0 5"
58010,"6RGW0"
57994,"eDP37"
57993,"HDMIA45"
57993,"Built-in display"
57993,"0x13f8"
57993,"0x00000000"
57991,"RIS 23""
57989,"LQ156Z1"
57988,"       "
57914,"iuaua{sv})"
55058,"HDMIA52"
53200,"n_idle_cb"
52306," @1 "
52280," n(U"
52247,"     "
52222,"AOC Intl 23""

As you can see, there a some strings that appear enormous amount of times, the most notable of them are the ones related to my monitors:
RD24L
i2352Vh
AOC Intl 23
Those, among other strings are parts of my monitors' model numbers - so it seems this has something to do with the monitor profiles, but here I'm not really sure, I'm not that of an expert on neither Gnome or it's monitor control routines.
Hope somebody here can help us with that.
Comment 13 Bastien Nocera 2016-10-02 13:08:01 UTC
I can't reproduce this problem here, but you might want to try and run gnome-settings-daemon under valgrind with the "--timed-exit" option, for it to exit cleanly. At the very least, gather a log and see whether some events occur regularly that could help pinpoint the culprit.

Other question might be whether you have screens color calibrated. As requested in comment 3 and 4, you should also try to disable some of the plugins, one each time. Don't forget to restart the daemon in between.
Comment 14 Michael Sandler 2016-10-07 16:24:02 UTC
Thanks for the tips on debugging the problem, here is what I got so far:

1. I've ran the replace gsd replace command again to look at its output (when I did it before I ran it win nohup), and indeed saw a lot of strings related to the color profile plugin, seems that there is a problem with the profile of one of my monitors, as you guessed. For some reason, 'colormgr get-devices' sees only one of my monitors, and is seems there is no profile attached to it, and therefore it is not calibrated.
To really calibrate it I understood I need a Windows/OSX installation to copy the ICC from, which I currently don't have, so I've just disabled the color plugin for now.
The repetitive logs of the daemon itself are in the attachment.

2. I've ran the daemon with the replace command using the following valgrind command:
$ valgrind --log-file=gsd.valgrind.log --leak-check=full --show-leak-kinds=all /usr/lib/gnome-settings-daemon/gnome-settings-daemon --replace
I ran this overnight, and I forgot the '--timed-exit' option you suggested, but it still generate a log, though for some reason it seems that it got overwritten when it reached 11M in size.
The log is attached too.
This command was ran while the color plugin was still on.

3. Once I disabled the color plugin, it seems that the problem was partially gone, but now it just takes more time for it to eat up memory.
Now also, the gsd daemon that was leaking under my user, is leaking a lot less, but the on under the 'gdm' user is still leaking massivly (Why are there two of them?).

To tell you the truth, it's kind of hard to tell for sure if disabling it did something to the leak, as numbers are still growing.
I'm going to try to install Windows on some flash drive to get an ICC profile, and see if that helps, that'll take some time.
Do you have any suggestions on how to give the daemon another run under valgrind with better options?
Also, the daemon that runs under the gdm user seems to be hard to debug, as the replace command replaces the one running under my user. Any insight on that?
Comment 15 Michael Sandler 2016-10-07 16:24:58 UTC
Created attachment 337180 [details]
debug logs
Comment 16 Bastien Nocera 2016-10-11 17:08:48 UTC
(In reply to Michael Sandler from comment #14)
<snip>
> 2. I've ran the daemon with the replace command using the following valgrind
> command:
> $ valgrind --log-file=gsd.valgrind.log --leak-check=full
> --show-leak-kinds=all /usr/lib/gnome-settings-daemon/gnome-settings-daemon
> --replace
> I ran this overnight, and I forgot the '--timed-exit' option you suggested,

With the timed-exit, you won't need to run it overnight.

Adding the "--track-origins=yes" argument would help get better information.

> but it still generate a log, though for some reason it seems that it got
> overwritten when it reached 11M in size.

Weird.

> The log is attached too.
> This command was ran while the color plugin was still on.
> 
> 3. Once I disabled the color plugin, it seems that the problem was partially
> gone, but now it just takes more time for it to eat up memory.
> Now also, the gsd daemon that was leaking under my user, is leaking a lot
> less, but the on under the 'gdm' user is still leaking massivly (Why are
> there two of them?).

There might be 2 problems in that log. The error path would be leaking memory, and colord isn't able to detect the device, in a loop.

The log you attached shows that it went up to 4 gigs of memory allocated, in one night. I guess that if, with the color plugin disabled, you don't get to that amount, then this is where the problem lies.

> Also, the daemon that runs under the gdm user seems to be hard to debug, as
> the replace command replaces the one running under my user. Any insight on
> that?

If you can reproduce the problem in your session, best not bother trying to debug the gdm one yet.

Additional question, what's the output of "xrandr -q" on that system?
Comment 17 Richard Hughes 2016-10-12 08:39:54 UTC
What seems to be happening is that the hardware is outputting a *lot* of primary-changed events. Each time this happens g-s-d queries colord for the icc profile as when the primary changes there's a high chance of a different crtc being used which means setting a new gamma table. As it's happening so frequently, I think it's the case that we're just stacking up the async events faster than they can be processed.
Comment 18 Vasily Khoruzhick 2016-10-19 01:06:40 UTC
I'm hitting the same issue. Here's the most suspicious part of valgrind output (gsd was running for few minutes), sorry for missing debug symbols:

==3347== 3,555,523 (15,920 direct, 3,539,603 indirect) bytes in 398 blocks are definitely lost in loss record 13,820 of 13,820
==3347==    at 0x4C29BBE: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3347==    by 0x68C7B98: g_malloc (in /usr/lib/libglib-2.0.so.0.5000.1)
==3347==    by 0x68E00D2: g_slice_alloc (in /usr/lib/libglib-2.0.so.0.5000.1)
==3347==    by 0x68FE28D: ??? (in /usr/lib/libglib-2.0.so.0.5000.1)
==3347==    by 0x68FAF23: g_variant_builder_end (in /usr/lib/libglib-2.0.so.0.5000.1)
==3347==    by 0x6352FEE: ??? (in /usr/lib/libgio-2.0.so.0.5000.1)
==3347==    by 0x6352DA1: ??? (in /usr/lib/libgio-2.0.so.0.5000.1)
==3347==    by 0x6355003: g_dbus_message_new_from_blob (in /usr/lib/libgio-2.0.so.0.5000.1)
==3347==    by 0x635F28C: ??? (in /usr/lib/libgio-2.0.so.0.5000.1)
==3347==    by 0x631C322: ??? (in /usr/lib/libgio-2.0.so.0.5000.1)
==3347==    by 0x631C358: ??? (in /usr/lib/libgio-2.0.so.0.5000.1)
==3347==    by 0x68C2439: g_main_context_dispatch (in /usr/lib/libglib-2.0.so.0.5000.1)
Comment 19 Vasily Khoruzhick 2016-10-25 01:08:03 UTC
Btw, for me one that leaks seems to be network manager:

strings core.826 | sort | uniq -c | sort -nr -t ',' -k 1,2 | head -n 42
  23767 org.freedesktop.NetworkManager
  11894 org.freedesktop.DBus.Properties
  11865 :1.3
  11744 org.freedesktop.NetworkManager.AccessPoint
   6916 DBus
   5910 PropertiesChanged
   5865 flags
   5864 hw-address
   5863 mode
   5860 wpa-flags
   5860 strength
   5860 ssid
   5860 rsn-flags
   5860 max-bitrate
   5860 last-seen
   5860 frequency
   5812 AccessPoint'
   5114 tworkManager'
   4688 desktop.DBus
Comment 20 Bastien Nocera 2016-10-25 08:41:38 UTC
If you're using Fedora 25, you might want to try to upgrade gnome-session and gnome-settings-daemon to 3.23 versions, available at:
http://koji.fedoraproject.org/koji/packageinfo?packageID=315
and:
http://koji.fedoraproject.org/koji/packageinfo?packageID=5615

The plugins in that version are stand-alone, which should make root-causing easier.
Comment 21 Vasily Khoruzhick 2016-10-26 17:38:53 UTC
(In reply to Bastien Nocera from comment #20)
> If you're using Fedora 25, you might want to try to upgrade gnome-session
> and gnome-settings-daemon to 3.23 versions, available at:
> http://koji.fedoraproject.org/koji/packageinfo?packageID=315
> and:
> http://koji.fedoraproject.org/koji/packageinfo?packageID=5615
> 
> The plugins in that version are stand-alone, which should make root-causing
> easier.

I'm using archlinux. However I believe that leaking plugin is 'sharing', I disabled it using dconf-editor and constant leak is gone.

But there's another - gsd leaks few megs each time when I plug/unplug external monitor.
Comment 22 Bastien Nocera 2016-10-26 17:42:29 UTC
Only thing I can advise would be to use gnome-settings-daemon and gnome-session 3.23.2, so you have each plugin split off into its own daemon, and debug those individually. The previous code is just impossible to debug properly with so many moving parts.
Comment 23 Vasily Khoruzhick 2016-10-26 23:57:29 UTC
(In reply to Bastien Nocera from comment #22)
> Only thing I can advise would be to use gnome-settings-daemon and
> gnome-session 3.23.2, so you have each plugin split off into its own daemon,
> and debug those individually. The previous code is just impossible to debug
> properly with so many moving parts.

Is gnome-3.22 compatible with gsd-3.23 and gnome-session-3.23? I don't want to compile whole gnome.
Comment 24 Bastien Nocera 2016-10-27 09:02:37 UTC
(In reply to Vasily Khoruzhick from comment #23)
> (In reply to Bastien Nocera from comment #22)
> > Only thing I can advise would be to use gnome-settings-daemon and
> > gnome-session 3.23.2, so you have each plugin split off into its own daemon,
> > and debug those individually. The previous code is just impossible to debug
> > properly with so many moving parts.
> 
> Is gnome-3.22 compatible with gsd-3.23 and gnome-session-3.23? I don't want
> to compile whole gnome.

As long as you have both gsd-3.23 and gnome-session-3.23, you should be fine, yes.
Comment 25 GNOME Infrastructure Team 2019-03-20 11:35:18 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/gnome-settings-daemon/issues/298.