GNOME Bugzilla – Bug 160993
Excessive malloc's for new terminals
Last modified: 2005-08-18 11:35:23 UTC
Each time I create a terminal with g-t, I get tons of memory allocations. Valgrind reports: ==22461== 1108880 bytes in 1660 blocks are still reachable in loss record 7309 of 7311 ==22461== at 0x3414AF01: calloc (in /usr/lib/valgrind/vgpreload_addrcheck.so) ==22461== by 0x34DD91E5: IA__g_malloc0 (gmem.c:154) ==22461== by 0x343FBB45: _vte_table_new (table.c:74) ==22461== by 0x343FBD48: _vte_table_addi (table.c:284) ==22461== by 0x343F8C6B: _vte_matcher_add (matcher.c:89) ==22461== by 0x3440C3A1: vte_terminal_set_emulation (vte.c:11036) ==22461== by 0x34415ECD: vte_terminal_init (vte.c:11374) ==22461== by 0x34D3EF57: IA__g_type_create_instance (gtype.c:1596) ==22461== by 0x34D291D3: g_object_constructor (gobject.c:1045) ==22461== by 0x34D2773A: IA__g_object_newv (gobject.c:942) ==22461== ==22461== ==22461== 2538400 bytes in 3800 blocks are still reachable in loss record 7310 of 7311 ==22461== at 0x3414AF01: calloc (in /usr/lib/valgrind/vgpreload_addrcheck.so) ==22461== by 0x34DD91E5: IA__g_malloc0 (gmem.c:154) ==22461== by 0x343FBB45: _vte_table_new (table.c:74) ==22461== by 0x343FBD48: _vte_table_addi (table.c:284) ==22461== by 0x343F8C6B: _vte_matcher_add (matcher.c:89) ==22461== by 0x3440C156: vte_terminal_set_emulation (vte.c:11062) ==22461== by 0x34415ECD: vte_terminal_init (vte.c:11374) ==22461== by 0x34D3EF57: IA__g_type_create_instance (gtype.c:1596) ==22461== by 0x34D291D3: g_object_constructor (gobject.c:1045) ==22461== by 0x34D2773A: IA__g_object_newv (gobject.c:942) ==22461== ==22461== ==22461== 3112880 bytes in 4660 blocks are still reachable in loss record 7311 of 7311 ==22461== at 0x3414AF01: calloc (in /usr/lib/valgrind/vgpreload_addrcheck.so) ==22461== by 0x34DD91E5: IA__g_malloc0 (gmem.c:154) ==22461== by 0x343FBB45: _vte_table_new (table.c:74) ==22461== by 0x343FBD48: _vte_table_addi (table.c:284) ==22461== by 0x343FBF6E: _vte_table_addi (table.c:202) ==22461== by 0x343F8C6B: _vte_matcher_add (matcher.c:89) ==22461== by 0x3440C156: vte_terminal_set_emulation (vte.c:11062) ==22461== by 0x34415ECD: vte_terminal_init (vte.c:11374) ==22461== by 0x34D3EF57: IA__g_type_create_instance (gtype.c:1596) ==22461== by 0x34D291D3: g_object_constructor (gobject.c:1045) Note *MEGABYTES* of malloc usage. Overall, I am seeing about 500 kb per terminal I create (acoording to vg).
Probably depends on background settings (translucent, image etc) and scrollback buffer. What are your settings?
I have no background crap (just a standard black-on-white terminal). My scrollback is fairly large, however these don't sound like they have anything at all to do with the scrollback. But even if I have 500 lines of scrollback, 500kb of allocation per terminal seems very silly (if my terminal is 100 chars wide, I would need at most 50 kb, assuming 1 byte per char).
Data structures are not optimal in table.c: struct _vte_table { GQuark resultq; const char *result; unsigned char *original; gssize original_length; int increment; struct _vte_table *table[_vte_table_max]; // _vte_table_max = 162 }; 668 bytes per tree node and about 574 nodes are created for a terminal. That's 383,432 bytes. And my intuition tells me that most of it is not used.
Created attachment 38273 [details] [review] proposed patch This patch saves about 260kb (383.432 bytes before patch, 120,800 bytes after patch).
Comment on attachment 38273 [details] [review] proposed patch Ups, memleak :(
Created attachment 38274 [details] [review] proposed patch Fixed memleaks of previous patch
Very, very nice looking patch. I'll have to try it out here. Is there any reason we can't just share this data structure between terminals? It seems excessive to create multiple versions of this.
FYI, konsole uses only 50kb per new terminal. I tested by creating 50 and looking at the diff in rss.
Created attachment 38337 [details] [review] one more leak fixed Run under Valgrind, looks fine now
Will this get into 2.10?
No, I don't think it will. However, I am going to push it for 2.10.1. Also, I am pushing it for SUSE 9.3. It should be going into a beta the deadline for which is tmw. It should get some extra testing there.
Created attachment 38481 [details] [review] Allows to share 'struct _vte_matcher' between terminal This patch adds a hash table and stores _vte_matcher for each terminal emulation there. So now it's possible to share one matcher instance among many terminals and one tab takes about 150kb (don't have exact number). Matcher is stateless, so it should not cause any problems. This patch does not depend on previous one. Run under Valgrind and used it for a while - so far it seems OK. Comments and suggestions are welcome ;)
By my count, we are using a bit less than 100 kb per tab now! Very, very nice.
Created attachment 38525 [details] [review] Shares 'struct _vte_matcher' and 'struct _vte_termcap' between terminals. Rewrote previous patch to use GCache instead GHashTable, it should be a bit cleaner now. Added similar cache for termcap data to save a bit of memory and to avoid reading and parsing the same file for each tab.
This patch should be reviewed for 2.10.1. Am getting distros shipping it. SUSE already is and Ubuntu is coming soon.
I commited the patch from comment #14. This is the only patch needed I reckon? Can the others be marked as obsolete?
attachment 38337 [details] [review] (from comment 9) should be commited too
As I understand it, the patch in comment 9 is now just a one time memory savings. If so, it is probably not worth it. The issue here is really how big things grow.
Yes, it's a one-time memory saving. But patch is pretty simple and it won't hurt anyone either. Slightly offtopic: gnome-terminal uses more than 10Mb RSS while xterm uses only 2.8Mb RSS. What's the number for konsole? It would be nice to make some more one-time savings and cut off few Mb.
Re. termcap caching, how about ripping off the builtin termcap parser altogether? See bug #169295.
I commited the patch from comment #9 too now so we can shake out any issues from CVS builds.
It isn't just new terminals that leak memory. I'm running Gnome-terminal 2.8.2 (Debian Sarge's current version -- and Sarge isn't even final yet) and had about ten windows with pretty large scrollback buffers. I'm on a 512 MB RAM machine (dual Athlon). Gnome-terminal burned up more than 321 MEGABYTES virtual, consuming more than half the available RAM in the system. This is obese, insane! I was running "script" in most of these windows, logging all the output; there isn't a typescript that's larger than 4MB (so if you assumed they were all 4MB, that's 40MB max of scrollback memory). In addition, gnome-terminal doesn't give that memory back to the system once you close those separate terminal windows. I closed 9 of the 10 windows, and the process may now (for all I know) have 200MB of free memory in it, but the kernel doesn't know it: here's the top listing: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2046 gnu 11 0 321m 319m 217m S 6.6 63.4 141:07.70 gnome-terminal At one time Emacs had a similar problem. As a long-lived process that will be used for days and days, and many windows, it needed to learn how to shrink its memory footprint as well as merely growing it. Stallman fixed it so that it allocates memory for each buffer in a separate arena, and unmaps that arena when the buffer is deleted, restoring the memory for use by other processes. Gnome-terminal could do that for each separate terminal window or tab.
what is the status of this bug? Can it be closed?
Yeah. This can be closed I guess. There are some other reports with patches to reduce memory consumption and those will go in soon.