GNOME Bugzilla – Bug 794690
QFX import matches nothing, crashes on corrections
Last modified: 2018-06-30 00:06:01 UTC
Created attachment 370136 [details] "Send to Apple" log QFX import of 17 items downloaded from a credit card account. All Descriptions match multiple historical transactions. All Info/allocations are wrong. Attempting to correct more than one results in a Gnucash crash.
Examples. "Spotify USA". 18 historical entries, all to the same account. Imports as "New, UNBALANCED" "DreamHost dh-fee.com". 20 historical entries, all to the same account. Imports as "New, UNBALANCED" "TMOBILE*AUTO PAY". 15 historical entries, all to the same account. Imports as "New, UNBALANCED" Different "WHOLEFDS" import as "Gas & Fuel" or as "Restaurants" when there are dozens of historical entries, all assigned to "Groceries".
Are you using Bayesian or direct matching? (See Preferences>Online Banking, under "Generic Importer", the 4th checkbox if you don't remember). Unfortunately the crash report doesn't tell me much: It actually looks like a Gtk problem. If you haven't run GnuCash again, please attach the tracefile, see https://wiki.gnucash.org/wiki/Tracefile.
Thanks John. I presume I'm using the default preferences, all three of the Generic Importer checkboxes checked including "Use bayesian matching". The docs don't say what happens if you don't use Bayesian matching; what is its alternative? I only recently upgraded from 2.6.16 to 2.6.19; the importer never crashed in nearly a year of using 2.6.16 and the matching was, well, better. I had to clear receipts off my desk and so I corrected everything manually for two imported accounts so a tracefile is not available. I'm setting a reminder for next month.
Created attachment 371244 [details] tracefile from beginning of import One month later. Trying to import a QFX file from a credit card provider.
Resorting to trying one week at a time for four weeks. Known wine bar purchase mapped to unrelated checking account, not an expense. Purchase of a banh mi mapped to purchase interest charge. Known restaurant purchases mapped to unrelated checking accounts or expense/groceries. Multiple new transactions with known vendors mapped to "Do not import". Too many discrepancies to list.
First attempt to match import crashes app. * 13:15:38 INFO <gnc.import> [gnc_imap_add_account_bayes] account name: 'Expenses:Personal:Food & Dining:Alcohol & Bars' * 13:15:38 INFO <gnc.import> [gnc_imap_add_account_bayes] adding token 'Wednesday' * 13:15:38 INFO <gnc.import> [gnc_imap_add_account_bayes] adding token 'PUNCHDOWN' * 13:15:38 INFO <gnc.import> [gnc_imap_add_account_bayes] adding token '*THE' * 13:15:38 INFO <gnc.import> [gnc_imap_add_account_bayes] adding token 'SQU*SQ' * 13:15:38 INFO <qof.engine> [qof_event_generate_internal] id=9 hi=0x3a077b0 han=0x1b4e40 data=0x0 * 13:15:38 INFO <qof.engine> [qof_event_generate_internal] id=8 hi=0x38b2e20 han=0x1b4e40 data=0x0 * 13:15:38 INFO <qof.engine> [qof_event_generate_internal] id=7 hi=0x12426f40 han=0x1b4e40 data=0x0 * 13:15:38 INFO <qof.engine> [qof_event_generate_internal] id=6 hi=0x11f9ff40 han=0x2a31b0 data=0x0 * 13:15:38 INFO <qof.engine> [qof_event_generate_internal] id=5 hi=0x11943160 han=0x304780 data=0x0 * 13:15:38 INFO <qof.engine> [qof_event_generate_internal] id=3 hi=0x3bef5c0 han=0x196c90 data=0x0 * 13:15:38 INFO <qof.engine> [qof_event_generate_internal] id=2 hi=0x3a68fb0 han=0x1f5f2d0 data=0x0 * 13:15:38 INFO <qof.object> [qof_object_foreach] type=Split * 13:15:38 INFO <qof.engine> [qof_collection_foreach] Hash Table size of Split before is 32381 * 13:15:38 INFO <qof.engine> [qof_collection_foreach] Hash Table size of Split after is 32381 * 13:15:38 INFO <qof.query> [qof_query_run_internal] matching objects=0x7c9c410 count=1875 * 13:15:38 INFO <qof.object> [qof_object_foreach] type=Split * 13:15:38 INFO <qof.engine> [qof_collection_foreach] Hash Table size of Split before is 32381 * 13:15:38 INFO <qof.engine> [qof_collection_foreach] Hash Table size of Split after is 32381 * 13:15:38 INFO <qof.query> [qof_query_run_internal] matching objects=0x8cca040 count=157 * 13:15:38 INFO <qof.object> [qof_object_foreach] type=Split * 13:15:38 INFO <qof.engine> [qof_collection_foreach] Hash Table size of Split before is 32381 * 13:15:38 INFO <qof.engine> [qof_collection_foreach] Hash Table size of Split after is 32381 * 13:15:38 INFO <qof.query> [qof_query_run_internal] matching objects=0x8cd2230 count=639 * 13:15:38 INFO <qof.engine> [qof_event_generate_internal] id=1 hi=0x3b7c890 han=0x2eced0 data=0x0
The same errors occur with 2.6.18. I am running OS X 10.13.4/High Sierra and cannot roll back to 2.6.16. Now I remember why I had to upgrade in the first place.
Import does not crash with 2.6.21, although matching appears here to be a popularity contest, e.g., assign everything (known & recurring public transit & Internet service providers) to Expenses/Groceries.
That sounds like your bayesian import maps have become corrupt in the process of upgrading somewhere :( Will need more in depth evaluation.
There are a few possible avenues to follow here. * Did you rename your accounts or restructured your account tree ? That would cause lots of invalid import map entries in 2.6. * The handling of import map data got rewritten for 3.0. In combination with this some backward compatibility code changes were done in the 2.6 series. These happened mostly between 2.6.18 and 2.6.19. I wonder if these 2.6 changes may have been buggy. It would be helpful to test this on an old version of your book that never got opened with 2.6.18. Instead open it in gnucash 2.6.16 and try to import one of your more recent qfx files. Another test would be to open such an old book in 2.6.21 only (ensuring it never got opened with 2.6.18 or 2.6.19). I read on the mailinglist though you have migrated to gnucash 3.1 which gives you an import map editor to help cleaning up. Please let us know how the importer behaves after this cleanup as well.
If you feel like closing this ticket it's fine with me. I did rename & restructure my account tree. Invalidating import rules seems a stiff penalty for what seems to me to be routine housecleaning and I'm glad it's no longer imposed in versions 3+. I will note two types of corruption that occurred. It seems very odd that account names would appear in the "Match String" column of the Import Map Editor. That is a very common occurrence in my map. It also seems odd that the same "Match String" is allowed to appear more than once with the same "Mapped to Account Name", e.g., Wednesday, Map Account NOT found, 1 Wednesday, Map Account NOT found, 1 I'm bringing database thinking to this but it seems this should have the equivalent of a uniqueness constraint on it and be coerced to Wednesday, Map Account NOT found, 2 In my case, anyway, matching on day names is going to cause more harm than good, as is matching on "in", or "of", or any of a number of other one and two character strings. But that is perhaps something to be discussed on the dev list.
(In reply to Eric Theise from comment #11) > If you feel like closing this ticket it's fine with me. > > I did rename & restructure my account tree. Invalidating import rules seems > a stiff penalty for what seems to me to be routine housecleaning and I'm > glad it's no longer imposed in versions 3+. > Yes, that was a bad bug :( And it's very likely the source of your original issue. > I will note two types of corruption that occurred. > > It seems very odd that account names would appear in the "Match String" > column of the Import Map Editor. That is a very common occurrence in my map. > > It also seems odd that the same "Match String" is allowed to appear more > than once with the same "Mapped to Account Name", e.g., > > Wednesday, Map Account NOT found, 1 > Wednesday, Map Account NOT found, 1 Before you restructured your account tree these would very likely have been mapped to distinct accounts. After restructuring the two account mappings got lost and both were (probably independently of one another) remapped to "Map Account NOT found". > > I'm bringing database thinking to this but it seems this should have the > equivalent of a uniqueness constraint on it and be coerced to > > Wednesday, Map Account NOT found, 2 > I don't know how easy that would be. This data is not bound to database concepts. It comes from the statistical analysis domain. > In my case, anyway, matching on day names is going to cause more harm than > good, as is matching on "in", or "of", or any of a number of other one and > two character strings. But that is perhaps something to be discussed on the > dev list. Perhaps. This is how the bayesian algorithm works. It takes all components of the data to match and applies weights to it. I don't know if this can be refined or not. I'm not a mathematician so I don't know enough of statistical analysis to judge this :( Feel free to bring it up on the devel list though. Perhaps people with more knowledge of that domain can add useful improvements indeed. I'm resolving this bug as "obsolete" because I want to acknowledge gnucash 2.6 behaved pretty badly in this area but - we won't release any 2.6 versions any more - 3.x should be affected much less by this (though I don't know what happens if you drop an account which is used in import map entries).
GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=794690. Please update any external references or bookmarks.