GNOME Bugzilla – Bug 585360
Monitor fontconfig configuration files using gio causes mtab ioctl storm
Last modified: 2009-06-24 11:41:52 UTC
Please describe the problem: The libxsetting.so since 2.25.1 uses g_file_monitor to monitor fonts changes. Essentially, everything that fontconfig fonts directories are monitored. These monitoring response to a mtab_file_changed event when /etc/mnttab is updated. for example, when I did a stop in g_unix_mount_at in dbx, it gives me after /etc/mnttab has been updated. (dbx) where current thread: t@1 =>[1] g_unix_mount_at(mount_path = 0x80dcc40 "/etc/fonts/conf.d", time_read = (nil)), line 1049 in "gunixmounts.c" [2] mounts_changed(mount_monitor = 0x80a0968, user_data = 0x8086858), line 186 in "glocaldirectorymonitor.c" [3] g_cclosure_marshal_VOID__VOID(closure = 0x80dcea0, return_value = (nil), n_param_values = 1U, param_values = 0x828c848, invocation_hint = 0x80474dc, marshal_data = (nil)), line 78 in "gmarshal.c" [4] g_closure_invoke(closure = 0x80dcea0, return_value = (nil), n_param_values = 1U, param_values = 0x828c848, invocation_hint = 0x80474dc), line 771 in "gclosure.c" [5] signal_emit_unlocked_R(node = 0x80de0f0, detail = 0, instance = 0x80a0968, emission_return = (nil), instance_and_params = 0x828c848), line 3248 in "gsignal.c" [6] g_signal_emit_valist(instance = 0x80a0968, signal_id = 105U, detail = 0, var_args = 0x8047610), line 2977 in "gsignal.c" [7] g_signal_emit(instance = 0x80a0968, signal_id = 105U, detail = 0, ... = 0xfe79e664, ...), line 3034 in "gsignal.c" [8] mtab_file_changed(monitor = 0x80b05b0, file = 0x829bf80, other_file = (nil), event_type = G_FILE_MONITOR_EVENT_CHANGED, user_data = 0x80a0968), line 1209 in "gunixmounts.c" [9] _gio_marshal_VOID__OBJECT_OBJECT_ENUM(closure = 0x80dce60, return_value = (nil), n_param_values = 4U, param_values = 0x80892a0, invocation_hint = 0x8047750, marshal_data = (nil)), line 202 in "gio-marshal.c" [10] g_closure_invoke(closure = 0x80dce60, return_value = (nil), n_param_values = 4U, param_values = 0x80892a0, invocation_hint = 0x8047750), line 771 in "gclosure.c" [11] signal_emit_unlocked_R(node = 0x80dd2c0, detail = 0, instance = 0x80b05b0, emission_return = (nil), instance_and_params = 0x80892a0), line 3248 in "gsignal.c" [12] g_signal_emit_valist(instance = 0x80b05b0, signal_id = 104U, detail = 0, var_args = 0x8047890), line 2977 in "gsignal.c" [13] g_signal_emit(instance = 0x80b05b0, signal_id = 104U, detail = 0, ... = 0x829bf80, ...), line 3034 in "gsignal.c" [14] emit_cb(data = 0x829b650), line 335 in "gfilemonitor.c" [15] g_idle_dispatch(source = 0x81f3678, callback = 0xfe747970 = &`libgio-2.0.so.0.1800.3`gfilemonitor.c`emit_cb(gpointer data), user_data = 0x829b650), line 4235 in "gmain.c" [16] g_main_dispatch(context = 0x808d028), line 2146 in "gmain.c" [17] g_main_context_dispatch(context = 0x808d028), line 2697 in "gmain.c" [18] g_main_context_iterate(context = 0x808d028, block = 1, dispatch = 1, self = 0x8091118), line 2778 in "gmain.c" [19] g_main_loop_run(loop = 0x82641d0), line 2986 in "gmain.c" [20] gtk_main(), line 1200 in "gtkmain.c" [21] main(argc = 1, argv = 0x8047a80), line 321 in "main.c" The problem here is that there 88 fonts related directories and that is multiplied by the number of entries in /etc/mnttab and the number of gnome-settings-daemon in a multi-user environments such as Sun Ray. In Sun Ray thing is particularly bad where user's home directory is often NFS mount and most user log in as such can trigger a change to /etc/mnttab which is a multiple of fonts dir and mnttab entries per user. As I investigating the problem, I wrote a sample program as that of fontconfig-monitor.c and it doesn't seems to have affected by an update of /etc/mnttab. However, it does as with gnome-settings-daemon. I have searched through all the modules of gsd, I couldn't find another use of g_file_monitor and similar. The stack trace shown also indicate that it is related to this module. I logged this bug to get your attention, behdad :) and hopefully your knowledge on this and also maintainer of gsd can shed more lights on this. THANKS. Steps to reproduce: 1. 2. 3. Actual results: Expected results: Does this happen every time? Other information:
Created attachment 136282 [details] the list of fonts directories that is being monitored on OpenSolaris
You'd have better chances if you actually CC'd him. ;-)
I'm not following what's the problem. Can you phrase it in one paragraph please?
I'm not a developer but how about I take a stab at it since, i'm seeing the behavior and led to Ghee's post. As more and more users login to a Sunray server, they mount their home directories via automount/nfs and /etc/mnttab gets updated. With each change to mnttab, each gnome-settings-daemon updates it's font list. With 10 users on the system, a SUN dual processor sparc server with 8gb is able to handle it, the load averages are about 2, put 30 sunray users on the same box and load averages go sky high (20 or more). So just one user login or logout on a server with 50-75 users causes an update to /etc/mnttab and 50-75 gnome-settings-daemons to update their font list at the same time.
Hi Behdad, I have dugged in a bit more. Let me try to rephrase so that you may be able to shed some light possibly from the gio perspective. 1. gsd set up fonts directory monitoring (there are 88 of such directories in OpenSolaris), 2. When /etc/mnttab, that triggers off a 'change' signal which called mtab_file_changed (gunixmounts.c in glib/gio) 3. Since the fonts directory has been registered, gio goes through every single entries registered checked and calls g_unix_mount_at() (gio/gunixmounts.c) which in turn reading every single entries of /etc/mnttab. 4. On Solaris, reading of /etc/mnttab is an ioctl call. 5. Currently, the gio implementation does not cache the /etc/mnttab entries, and hence, every single entry costs an ioctl call. Typically, /etc/mnttab would have around 100 entries. 6. so, a update of /etc/mnntab costs: Number fonts dir x N(mnttab) x number of gsd I have to say now, this is not a gsd bug. but a gio defect. A serious one for GNOME on Sun Ray. For example, I know that gnome-panel also has directory monitoring for desktop file, though that would be of a smaller number. Any thought or suggestion as to how best to approach this, either from the gsd perspective of gio perspective? Thanks, -Ghee
Definitely gio bug. It should cache the contents at least, and perhaps do randomized delay before scanning.
I appreciuate everyones help on this. From a SUN Ray admin perspective, I don't understand why the fonts need to be scanned after a user logs in. We never add new fonts, so for us never scanning for fonts after a user logs in would work best.
Fonts are not rescanned. It's gio trying to see if any files being monitored changed as a result of new mounts. That's a gio issue. For disabling font monitoring completely, I filed bug 585455.
Why is the local code being used here anyway ? Should this not use the fen code in your case ?
(In reply to comment #9) > Why is the local code being used here anyway ? Sorry, what do you meant by local code? > Should this not use the fen code in your case ? Lin, have you any idea who do we manage fen in Solaris? >
> > Should this not use the fen code in your case ? > > Lin, have you any idea who do we manage fen in Solaris? > I do. But fen code doesn't monitor /et/mnttab. It monitors whatever the application requests. I think in this case there are redundant GMount calls, I think we should fix it in the following sides: 1) fix from Solaris kernel, do not update /etc/mnttab if possible or provide efficient functions for accessing the contents of mnttab. 2) I'm guessing GMount is used internally in gio/gvfs, but it should be used only on a request of an application. 3) Cache mnttab contents in gvfs.
Lots of confusion here. Basically, when the set of mounts change we catch the "mounts-changed" signal in the directory monitor. This then tries to determine if the monitored directory was affected by the mount/unmount. It does so by getting the GUnixMount for the monitored pathname, then compares it with the old one. If they differ we send an "unmount" event on the monitor. The problem is that getting the GUnixMount for a pathname requires reading /etc/mnttab (or equivalent). So, this is a performance problem. Fortunately the fix is easy, we actually have a flag you need to pass in if you want mount change notification (G_FILE_MONITOR_WATCH_MOUNTS) so we just need to check this and avoid doing this unnecessarily.
(In reply to comment #12) > Fortunately the fix is easy, we actually have a flag you need to pass in if you > want mount change notification (G_FILE_MONITOR_WATCH_MOUNTS) so we just need to > check this and avoid doing this unnecessarily. So if G_FILE_MONITOR_WATCH_MOUNTS is not set, we implied that watch mount is no necessary right? since the fonts monitoring is gnome-settings-daemon is in http://git.gnome.org./cgit/gnome-settings-daemon/tree/plugins/xsettings/fontconfig-monitor.c#n61 is for local file system and is to G_FILE_MONITOR_NONE, and that is where most cases are I think. Where should G_FILE_MONITOR_WATCH_MOUNTS be checked? Thanks!
I commited a fix for this in master, can you verify that this helps: http://git.gnome.org/cgit/glib/commit/?id=740ae3aa29cc50f588fb3ac47f6125acbccd20b6 If so we should backport it.
(In reply to comment #14) > I commited a fix for this in master, can you verify that this helps: > > http://git.gnome.org/cgit/glib/commit/?id=740ae3aa29cc50f588fb3ac47f6125acbccd20b6 > I tested the patch on glib/gio 2.18.3, it WORKS GREAT! no more cll to open /etc/mnttab > If so we should backport it. please :) > Karl, if you let me know what version of snv you are using, I see if I can build the right version of the libraries for you to test out in your environment. x86 or sparc. Thanks!
Backported to glib-2-20.
Can somebody explain the impact of disabling watch mount monitoring? Where/how is G_FILE_MONITOR_WATCH_MOUNTS set? Did anyone consider an approach with a system daemon that monitors the changes and uses a lightweight communication mechanism to the user sessions that need the information?
(to clarify, I'm asking about user experience or sys admin impact)
try out keyword field
G_FILE_MONITOR_WATCH_MOUNTS is set when you want to get events for the monitored directory getting unmounted. Its generally only used when e.g. displaying the directory in a file manager or file selector (where you want to close the window or suchlike when the directory is unmounted) I did not consider a complicated daemon approach for two reasons: a) This should be quite seldomly used b) On linux inotify gives events for unmount anyway so no need to do the complicated stuff there