GNOME Bugzilla – Bug 794644
Remove Old Prices is very slow
Last modified: 2018-06-30 00:05:52 UTC
Removing old prices in a file with lots of prices takes an extremely long time. So far it's been running 28 hours at 100% processor utilization. Of course there is no way to know how much longer it will run. This file has 144903 prices in it, which is a lot, but still this is extreme. The problem seems to be that every time a price is deleted, gnc_tree_model_price_do_deletions deletes the price from the tree model. The GTK tree model code uses gnc_tree_model_price_iter_next to loop over all prices to find the one being deleted. This code uses an N**2 algorithm to find the price. There are various ways this iterator could be made better, but perhaps the obvious fix is to not update the model during this process, but rather reconstruct it from scratch when it is done. I don't know if this is practical or not. I tried calling gnc_suspend_gui_refresh before starting the price deletion. This may have helped some, but it's still taking days to run.
Well that's a lot of prices!! If you are up to it can you make these changes to dialogprice-edit-db.c Add these lines... 390, GtkTreeModel *model; 463 // disconnect the model to the price treeview model = gtk_tree_view_get_model (GTK_TREE_VIEW(pdb_dialog->price_tree)); g_object_ref (G_OBJECT(model)); gtk_tree_view_set_model (GTK_TREE_VIEW(pdb_dialog->price_tree), NULL); 507, // reconnect the model to the price treeview gtk_tree_view_set_model (GTK_TREE_VIEW(pdb_dialog->price_tree), model); g_object_unref(G_OBJECT(model)); It works for me but I do not have so many prices, will need to import a lot to get to your numbers. Bob
Yes, it's a lot of prices. That's why I decided to purge some. I've been getting price quotes daily (mostly) since at least 2003. Your changes seemed to do the trick. I didn't realize it was that easy to reconstruct the model after the purge was done. Before this change I eventually killed GnuCash after it had spent 44:49:45, nearly two days, of processor time and not finished. After the change I opened my file and purged prices in a total of 11:31, of which about 3 minutes was the actual purge. Opening the price editor dialog before the purge took longer. The number of prices was reduced to 38268 and the total file size was reduced substantially. Shall I push this change? I'm not an expert on Gtk3 so Im not really the one to sign off on it.
I've already merged it. Thanks, Bob.
I think the same change should be made in gnc_prices_dialog_get_quotes_clicked. Getting price quotes takes about 45 minutes after purging old prices. At least 35 of those were spent after the Perl script finished and it was adding the new prices that had been returned. As a point of interest, about 95% of that time was spent in compare_prices_by_date called from gnc_pricedb_nth_price. Almost all of that time is in guid_compare since there are many prices with the same time stamp and compare_prices_by_date compares the GUIDs to get a stable sort. If we had some other way to get a stable sort this would save a lot of time. Or perhaps a stable sort isn't necessary thre. I would try some of these changes, but unfortunately I'm not in a position to do much with GnuCash right now. I hope this changes in a couple of months.
It would be better to open a new bug. Are you testing with 2.7.8 or better yet unstable HEAD? I ask because GUID::operator==, which would be called half the time in guid_compare, was using std::mismatch byte-by-byte on the two guids byte reprentations instead of the provided boost::uuid::operator==. I fixed that about a month ago.
I'm running 2.7.8 (the Mac version downloaded from gnucash.org). The time is spent in qof_instance_get_guid, not in the actual comparison. I haven't looked, but I suspect that qof_instance_get_guid is spending much of the time validating its argument. You're probably right that a new bug would have been appropriate.
qof_instance_get_guid's validation is just QOF_IS_INSTANCE, which just compares the GObject's type field with whatever QofInstance's GType value is. It's a straight int compare and should be about 3 machine instructions. Then it does GET_PRIVATE and returns &priv->guid. 3 more instructions. Something's not right there. When I profile stock retrieval using Instruments time profiler I don't get anything for qof_instance_get_guid or guid_compare, but then I pruned my online quoted stocks to get the Alphavantage retrieval time down. Could you save and attach your profiling data?
I had a look at gnc_prices_get_quotes_clicked. It doesn't have the necessary parameters to remove and rebuild the tree model, it just calls a scheme routine. It'll take a bit of work to figure out the right place.
Created attachment 370402 [details] Sample of where time is spent while fetching quotes
I had a long detailed description of the time sample I just attached, but BugZilla lost it. I didn't realize that attaching a file would trash my unsaved comment. I don't have time to recreate it now. The file contains the relevant parts of two samples taken during one price quote fetch. I'm sorry I lost the further description, I think it would have been useful.
GnuCash bug tracking has moved to a new Bugzilla host. The new URL for this bug is https://bugs.gnucash.org/show_bug.cgi?id=794644. Please continue processing the bug there and please update any external references or bookmarks.