GNOME Bugzilla – Bug 120028
Poor performance and mem usage with many accounts
Last modified: 2018-06-29 20:36:08 UTC
If you create an account structure with many accounts, GnuCash becomes unbelievably slow to open an account register window, and the memory usage increases dramatically with each open account register. Note, the presence or absence of transactions is irrelevant. In the attached example GnuCash file, there are 600+ accounts. This is the by-law mandated account structure for Brazilian businesses, and thus this is a real problem for business adoption in Brazil. Of course, this issue affects anyone who uses a lot of accounts. Using the graphical memory profiler "memprof" (comes in RH9, see www.gnome.org/projects/memprof/) I have tracked down some of the biggest RAM using routines. When you first open the attached file, GNC uses 10.6 MB RAM, after opening 5 empty register windows, GNC uses 22MB. Remember, there are no transactions! Following the "self" sizes as we open more register windows (#), below is shown the function using the most memory. # function size (MB) 0 lmod 3.4 1 lmod 3.4 2 lmod 3.4 3 gnc_quickfill_new 4.7 4 gnc_quickfill_new 6.3 5 gnc_quickfill_new 7.9 As you can see, gnc_quickfill_new is getting larger and larger as more register windows are opened. Following the "total" sizes as we open more register windows (#), below is shown the function using the most memory: # function size (MB) 0 gnc_module_load_common 14.8 1 gnc_module_load_common 14.8 2 __libc_start_main 15.6 3 __libc_start_main 18.0 4 __libc_start_main 20.4 5 __libc_start_main 22.8 So, here libc_start_main keeps growing and growing. Given the mane of this function, it doesn't look too helpful in identifying the offender. Oh well. I'm looking into setting up a "time in function" profile. I'll post here when I have it.
Created attachment 19266 [details] Example GNC file with 600+ accounts
FWIW: my initial message to the list about this: http://www.mail-archive.com/gnucash-devel@lists.gnucash.org/msg03265.html
As an experiment, you can start a new GNC file using the account setup druid, and select all. This creates a lot of accounts, 113 in fact. With this setup, the account register windows open quickly. Running this through memprof, shows that GNC uses 10MB initially, and only 12MB after opening 10 register windows. Even with the 10 register windows open, profiling the memory usage shows that both __libc_start_main and gnc_quickfill_new are relatively low emmory consumers. So, this is not a problem with account structures with less than 113 accounts.
x
The quickfil is not only a memory hog, its a cpu hog. Here's an execution time trace: Clock 2 Start: regWindowSimple: start reg window simple clock Clock 7 Start: gnc_ledger_display_interna...(): the begingi Clock 7 Elapsed: 0.009814s gnc_ledger_display_interna...(): just before query Clock 7 Elapsed: 0.010016s gnc_ledger_display_interna...(): just after query Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.188528s gnc_load_xfer_cell: post quickfil combo Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.174336s gnc_load_xfer_cell: post quickfil combo Clock 7 Elapsed: 0.380196s gnc_ledger_display_interna...(): the end Clock 2 Elapsed: 0.380539s regWindowSimple: post display_simple Clock 6 Start: regWindowLedger: start reg window ledger Clock 6 Elapsed: 0.269979s regWindowLedger: post initial setup Clock 6 Elapsed: 0.270108s regWindowLedger: post query Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.280238s gnc_load_xfer_cell: post quickfil combo Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.255263s gnc_load_xfer_cell: post quickfil combo Clock 6 Elapsed: 1.232379s regWindowLedger: the end Clock 2 Elapsed: 1.613017s regWindowSimple: fin with regWindowSimple The 4 calls to gnc_load_xfer_cell account for more than half of the total time. I'm not sure why there are 4.
Every time a new transaction is entered, the account-selection combo box is filled out four time. I'm not sure why. It should have been filled out once, when the ledger window is created. Here's a typical per-transaction record: Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.270136s gnc_load_xfer_cell: post quickfil combo Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.274058s gnc_load_xfer_cell: post quickfil combo Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.268635s gnc_load_xfer_cell: post quickfil combo Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.265186s gnc_load_xfer_cell: post quickfil combo
The 4 times through are because there are four quickfill cells in the register. There's the action, description, account, and something else... that's why four. As for why it's being re-loaded at every commit... Well, that's a good question.
The four calls are due to gnc_load_xfer_cell() called twice by gnc_split_register_load_xfer_cells() which is called by gnc_split_register_load() in register/ledger-core/split-register-load.c These calls should have been protected by if (info->first_pass) so that they'd be called only when the register is first opened, instead of each transaction update. I just moved these to a more correct location in that routine, and surrounded by the fisrt_pass test, and it seems to make a huge difference, and it seems to still work correctly. Will check in changes shortly. Am also investigating some other questionable timings.
Quickfill for ordinary cells is not a problem at all. The problem is with loading the zillion account names into the transfer-account combo-box. Since the list of accounts is *almost* static, we might be able to load the combobox once, and cache it for subsequent register opens. Maybe. I'm now trying to determine if its really the quickfill loading, or some other aspect of the combobox that so slow.
Yep, if we don't load the quickfill, then creating a combobox for 600 accounts takes a fortieth of a second: Clock 3 Start: gnc_load_xfer_cell: pre quickfil combo Clock 3 Elapsed: 0.022563s gnc_load_xfer_cell: post quickfil combo I'm writing some code to use a shared/cached quickfill for account account lists.
Note that you need to re-stuff the list whenever and account is created, destroyed, or renamed. So you do need to keep that in mind. I'll note that the business-ledger code probably wants to be similarly fixed, except in THAT case you can't quite use a global quickfill because the business register limits the accounts available to certain types based on the type of register.
OK, I've just checked in code to the CVS HEAD for src/register/ledger-core/split-register-load.c that builds and uses a quickfill cache for the account-transfer combo box. The first register open will still be slow, but all subsequent opens should be a lot faster. Some quick test show the memory growth problem is mostly alleviated by this (though not completely solved). To finish this defect, this code needs to be backported to the 1.8 branch.
The code should also be ported to the business ledgers. src/business/business-ledger/*Load.c
Damn, the code I checked in worked when I tested it, but now on retest, it core-dumps. Arghhh.
Fixed core dump; it was introduced by some last-minute changes. CVS HEAD now works again.
Back-ported the patch to speed transaction commit to the 1.8 tree. The patch to speed register open has not been backported. Please test, I'm getting a little cross-eyed.
Back-ported the register-open speed up too. Am marking this bug fixed-awaiting-test, close if it tests good.
Unfortunately 1.8 doesn't build: account-quickfill.c:37: warning: no previous prototype for `gnc_quickfill_get_string_match_mb' make[5]: *** [account-quickfill.lo] Error 1 I am re-opening this bug.. But I'll work on getting it to work.
Ok, it was just a missing "static". It's building now. I'll go test and make sure it's all working correctly.
Nope, I got it to crash on my first attempt. I opened an existing register, started to add a new transaction, wanted to create a new account, so I typed "ex" which quickfilled into my expenses, and then finished in Expenses:Anything. As soon as I tabbed out of the account cell it crashed with a SEGV: 0xffffe002 in ?? () (gdb) where
+ Trace 46984
$2 = {menustrings = 0x0, sheet = 0x84f0340, item_edit = 0x84f1d10, item_list = 0x850ac58, signals_connected = 1, list_in_sync = 1, list_sorted = 1, list_popped = 1, autosize = 0, qf = 0x8483ef8, use_quickfill_cache = 1, in_list_select = 0, strict = 1, complete_char = 58 ':', ignore_strings = 0x836fcb8} (gdb) p *box->qf $3 = {text = 0x0, len = 138936430, matches = 0x10} (gdb) So clearly something is just wrong... I'll commit the one change I've made, maybe you can look at this? I'll try HEAD to see if that's working any better.
Nope, HEAD crashes in the same way...
I committed a change which I *think* should fix this, but I haven't tested it yet. Let me understand the scenario: you tabbed to "Expense:" then created an account called "Anything", then tabbed to that?
Basically, yes. I tabbed over to the account cell, effectively typed in "Expenses:Anything", tabbed out, when it asked me if I wanted to create the account I said "yes", at which point it crashed. It doesn't crash anymore. HOWEVER.. If I have a register open, I then (from the main account-tree menu) click on "New Account". I then create a new account. Then I go back into the register but the new account isn't in the list. So the quickfill isn't actually handling new accounts (or modified accounts) correctly. This testing was in 1.8, not HEAD, but I presume it's the same issue in both places.
I've changed gnucash in 1.9.x to share more data between the "account" cells in different registers. Gnucash seems very quick now, even with 15 registers open at once.
IIRC the problem wasn't the number of open registers, but the number of accounts in the tree and the amount of time it took to refill the Transfer Cell in the register on every redraw/reload.
David- Did you try to reproduce this problem? I actually put a lot of work into reporting this bug, including attaching an example file and giving explicit instructions on how to measure the problem. It seems premature to me that you closed this bug with "Gnucash seems very quick now, even with 15 registers open at once". Thanks for all the work you are doing, I really appreciate it.
I did not test your exact scenario until now, but did my work based on Derek and Linas's comments and my own observations of how much gnucash slowed down with multiple open registers. The problem I fixed was an O(2n*m) problem where 'n' is the number of open registers and 'm' is the number of accounts in the COA. With the exception of the first register opened, that code is now O(1). I set up a test this morning for your scenario using your example accounts file. The time taken to open the 15th account register before my changes was 3.35 seconds. This is based on capturing the time at the start and end of the open function with calls to gettimeofday(). I took ten samples, tossed the highest and lowest, and averaged the remaining numbers. After my changes the time to open the 15th register is 0.17 seconds, calculated with the same method. This is a 95% performance improvement. I have not measured the memory impact, but I would expect it to be significantly improved as well. When you opened this bug there were 2 pairs of data structures created for for the "account quickfill" cells for each open account register (for a total of 4n data structures). Linas and Derek lowered that to one data structure shared across all open registers, and one pair of data structures per open register (total 2n+1). There is now one pair of account quickfill data structures shared across all open registers (total 2).
David, Just make sure that the standalone quickfill is listening to events so it gets notified of account changes, (name changes, new/deleted accounts, etc) so it can update the list. There was a bug in 1.8 where if you had an open account register and then created a new acccount, the new account wouldn't get into the list.
That's bug 302682. My changes use the existing listen_for_account_events() function so will have the same issue (which looks relatively easy to fix).
GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=120028. Please update any external references or bookmarks.