GNOME Bugzilla – Bug 592308
gnome-panel has severe memory leak or similar when software RAID scrub running
Last modified: 2020-11-06 20:20:17 UTC
gnome-panel process size grows to gigabytes when running a Linux md RAID sync_action check on the array that hosts /home (or possibly /usr, /var/spool, /opt, or /usr/local). To reproduce: 1. /home in a LVM Logical Volume in a Volume Group in a Physical Volume on a Linux md RAID1 array (e.g. /dev/md3) 2. Log into a GNOME session that includes gnome-panel 3. As root, echo check >/sys/block/md3/md/sync_action 4. Observe as gnome-panel RSS and VSZ rockets upwards Reproduced with kernel-2.6.29.6-217.2.3.fc11.x86_64 and gnome-panel-2.26.3-1.fc11.x86_64 but also occurs with earlier versions of both shipped with FC11. Not present in FC10, to the best of my knowledge. Also filed as Fedora Bug 486764
Also consumes a large amount of CPU time whilst scrub is running, though this returns to normal once scrub has completed.
FYI, hardware is 2*Seagate ST31000333AS 1TB SATA discs running fixed SD1B firmware connected to Intel SATA ports of an Asus P5Q (P45 chipset) motherboard. BIOS is configured to present SATA ports as AHCI rather than legacy devices.
OK, this might be a bit harder to reproduce than I first thought. I created another user ('clean') and tried to reproduce the problem and couldn't. I even added many of my normal user's panel applets with no success. I then moved the following directories out of the way in my normal user's home directory, so I'd start with the default FC11 GNOME setup: ./Desktop ./.xine ./.metacity ./.qt ./.gnome ./.config ./.nautilus ./.gnome2_private ./.gvfs ./.gtk-bookmarks ./.gfconf ./.dbus ./.gstreamer-0.10 ./.pulse-cookie ./.mcoprc ./.themes ./.pulse ./.gconf ./.gnome2 ./.icons ./.gstreamer-0.8 ./.kde ./.gtkrc My normal user was able to reproduce the problem. I also noticed that the 'clean' user had no SELinux contexts on their home directory. I used chcon's --reference switch to copy my normal user's contexts but still wasn't able to reproduce the problem when that user had a GNOME session running. I've tried running valgrind and strace on gnome-panel but I could do with a better idea of what I'm looking for, or maybe a more refined command line for those tools. Strace suggests gnome-panel is particularly interested in the following files as it repeatedly repeatedly tries open/access/stat them: /home/normal/.gtk-bookmarks /home/normal/.local/share/app /home/normal/.local/share/applications/gnome-beagle-search.desktop /home/normal/.local/share/applications/gnome/beagle-search.desktop /home/normal/.local/share/applications/gnome/beagle/search.desktop /home/normal/.local/share/applications/gnome-nautilus-computer.desktop /home/normal/.local/share/applications/gnome/nautilus-computer.desktop /home/normal/.local/share/applications/gnome/nautilus/computer.desktop /home/normal/.local/share/applications/gnome-network-scheme.desktop /home/normal/.local/share/applications/gnome/network-scheme.desktop /home/normal/.local/share/applications/gnome/network/scheme.desktop /usr/local/share/applications/gnome-beagle-search.desktop /usr/local/share/applications/gnome/beagle-search.desktop /usr/local/share/applications/gnome/beagle/search.desktop /usr/local/share/applications/gnome-nautilus-computer.desktop /usr/local/share/applications/gnome/nautilus-computer.desktop /usr/local/share/applications/gnome/nautilus/computer.desktop /usr/local/share/applications/gnome-network-scheme.desktop /usr/local/share/applications/gnome/network-scheme.desktop /usr/local/share/applications/gnome/network/scheme.desktop /usr/share/applications/gnome-beagle-search.desktop /usr/share/applications/gnome-nautilus-computer.desktop /usr/share/applications/gnome-network-scheme.desktop /usr/share/applications/.order
Created attachment 141888 [details] gnome-panel wrapper script which runs it under valgrind using --tool=massif
Created attachment 141889 [details] valgrind output at start
Created attachment 141890 [details] valgrind output after gnome-panel has used ~1GB, and valgrind is using ~2.6G
Created attachment 141891 [details] valgrind output at start (compressed)
From ms_print of .2017 MB 945.1^ ,, | . @# | . @ : @# | , @: @ : @# | ., : @ @: @ : @# | ,.::@ : @ @: @ : @# | . : @:::@ : @ @: @ : @# | .,: : @:::@ : @ @: @ : @# | . :::@: : @:::@ : @ @: @ : @# | . : : :::@: : @:::@ : @ @: @ : @# | . . :: : : :::@: : @:::@ : @ @: @ : @# | .. :: : :: : : :::@: : @:::@ : @ @: @ : @# | .. ::: :: : :: : : :::@: : @:::@ : @ @: @ : @# | . ::: ::: :: : :: : : :::@: : @:::@ : @ @: @ : @# | .: @: ::: ::: :: : :: : : :::@: : @:::@ : @ @: @ : @# | ...@ : :: @: ::: ::: :: : :: : : :::@: : @:::@ : @ @: @ : @# | . :::@ : :: @: ::: ::: :: : :: : : :::@: : @:::@ : @ @: @ : @# | ,:: :::@ : :: @: ::: ::: :: : :: : : :::@: : @:::@ : @ @: @ : @# | :@ @:: :::@ : :: @: ::: ::: :: : :: : : :::@: : @:::@ : @ @: @ : @# | , ::@ @:: :::@ : :: @: ::: ::: :: : :: : : :::@: : @:::@ : @ @: @ : @# 0 +----------------------------------------------------------------------->Gi 0 820.4 [...] -------------------------------------------------------------------------------- n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B) -------------------------------------------------------------------------------- 0 0 0 0 0 0 1 11,868,089,880 20,467,264 17,180,450 3,286,814 0 2 27,162,195,501 46,929,920 39,408,874 7,521,046 0 83.97% (39,408,874B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->45.70% (21,445,509B) 0x556B1E1: g_malloc (gmem.c:131) | ->34.95% (16,401,104B) 0x55804B0: g_slice_alloc (gslice.c:824) | | ->20.71% (9,720,256B) 0x55807B4: g_slice_alloc0 (gslice.c:833) | | | ->20.61% (9,671,596B) 0x5312099: g_type_create_instance (gtype.c:1654) | | | | ->20.53% (9,632,460B) 0x52F5F8A: g_object_constructor (gobject.c:1338) | | | | | ->20.49% (9,617,428B) 0x52F699C: g_object_newv (gobject.c:1215) | | | | | | ->19.27% (9,042,748B) 0x52F7523: g_object_new_valist (gobject.c:1278) | | | | | | | ->19.27% (9,042,748B) 0x52F767A: g_object_new (gobject.c:1060) | | | | | | | ->17.65% (8,281,328B) 0x43BDAE: panel_load_menu_image_deferred (menu.c:1880) [...] 76 879,150,505,223 988,960,256 822,311,092 166,649,164 0 77 879,458,837,690 989,368,504 822,648,807 166,719,697 0 83.15% (822,648,807B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->52.65% (520,864,578B) 0x556B1E1: g_malloc (gmem.c:131) | ->43.22% (427,602,268B) 0x55804B0: g_slice_alloc (gslice.c:824) | | ->26.37% (260,886,860B) 0x55807B4: g_slice_alloc0 (gslice.c:833) | | | ->26.36% (260,838,200B) 0x5312099: g_type_create_instance (gtype.c:1654) | | | | ->26.36% (260,797,600B) 0x52F5F8A: g_object_constructor (gobject.c:1338) | | | | | ->26.36% (260,782,568B) 0x52F699C: g_object_newv (gobject.c:1215) | | | | | | ->25.26% (249,873,400B) 0x52F7523: g_object_new_valist (gobject.c:1278) | | | | | | | ->25.26% (249,873,400B) 0x52F767A: g_object_new (gobject.c:1060) | | | | | | | ->25.23% (249,619,040B) 0x43BDAE: panel_load_menu_image_deferred (menu.c:1880) | | | | | | | | ->25.23% (249,616,576B) 0x43BF37: setup_menu_item_with_icon (menu.c:1763) | | | | | | | | | ->09.87% (97,675,776B) 0x44A83C: panel_place_menu_item_append_local_gio (panel-menu-items.c:659) | | | | | | | | | | ->09.87% (97,675,776B) 0x44ADF4: panel_place_menu_item_create_menu (panel-menu-items.c:1039) | | | | | | | | | | ->09.87% (97,674,192B) 0x44BB2D: panel_place_menu_item_recreate_menu (panel-menu-items.c:1086)
Hope the valgrind is useful. If not, please can someone get in touch to tell me what kind of debug info I should be providing. I just want this fixed! :-)
Created attachment 141956 [details] [review] debug patch for gnome-panel
After applying the above debug patch, I found that gnome-panel seemed to be excessively concerned with some unmounted cifs mountpoints that are in my normal user's home directory and listed in /etc/fstab. Moving these out of my normal user's home directory appears to prevent gnome-panel from eating memory when the RAID scrub is running. This will do as a workaround, but I'm not particularly happy about the intolerance for underlying UNIX functionality.
I'm seeing the same effect here; the gnome-panel process grows steadily in size to multiple gigabytes of ram while software raid devices are being checked. For reference I do have (currently unmounted) cifs mountpoints under /home and in /etc/fstab if that proves to be relevant. Name : gnome-panel Relocations: (not relocatable) Version : 2.30.0 Vendor: Fedora Project Release : 4.fc13 Build Date: Wed Jun 30 16:23:53 2010 After killing the old gnome-panel (which reached multiple gigabytes of ram) 3 hours ago: 7011 wstearns 20 0 772m 508m 8764 R 11.6 6.4 6:18.44 gnome-panel
2.5 hours later, that process is up to 1.8GB virtual, 1.5GB resident: 7011 wstearns 20 0 1771m 1.5g 6768 R 23.5 18.8 33:00.36 gnome-panel
It must be thursday morning, gnome-panel is growing again because of the raid validation running in the background: 2315 wstearns 20 0 2739m 2.4g 7596 S 39.4 15.1 76:21.41 gnome-panel
Created attachment 188425 [details] [review] Potential fix Okay, there was definitely something that looked wrong. Can you try this patch and see if it helps?
bugzilla.gnome.org is being replaced by gitlab.gnome.org. We are closing all old bug reports in Bugzilla which have not seen updates for many years. If you can still reproduce this issue in a currently supported version of GNOME (currently that would be 3.38), then please feel free to report it at https://gitlab.gnome.org/GNOME/gnome-panel/-/issues/ Thank you for reporting this issue and we are sorry it could not be fixed.