GNOME Bugzilla – Bug 653715
General Transaction matcher drops transactions
Last modified: 2018-06-29 22:59:36 UTC
Created attachment 190989 [details] sample gnucash file containing a single transaction The general transaction matcher will propose multiple possible matches for a single transaction already in the register. The user has no way of knowing that that both 'matches' point to the same existing transaction. If the user says "great, all the transactions from the bank match" and clicks OK, then the matcher picks one of the potential matches and silently discards the other(s). Attached are 2 files: MatcherTest.gnucash is a gnucash data file with several accounts and a single transaction. The file similartrans.ofx is a credit card ofx file with 2 transactions within a dollar for the amount and occurring one day apart (bank's view) and posting 3-4 days after the single transaction in the data file. Open MatcherTest.gnucash Set Edit>Preferences>Online Banking: Commercial ATM fees Threshold to 1.00 Enable the "Enable Update Match Action" (an abomination, but that's for another bug). Use File>Import>Import OFX/QFX to import the similartrans.ofx file. The general transaction matcher window appears with both transactions from the ofx file indicated as a good match to something in the data file. Click OK Gnucash accepts the first transaction as a match, marking the credit card split of the existing transaction as cleared, and throws away the second transaction from the matcher window/ofx file. If you import the same ofx file again, gnucash will present the transaction it had discard, indicating that it is a match for a (the only) transaction in the data file. If you click OK now, gnucash will swap the second transaction for the first (thereby silently deleting a transaction that had already been marked as cleared). You can swap these transactions at will by reimporting the ofx file as many times as you'd like, but unless you catch on and check the "A" box in the transaction matcher window, you will always be discarded valid data.
Created attachment 190990 [details] ofx file containing 2 similar transactions
That IS the expected behavior of the update match action. And the reason it can be disabled...
If you disable update match action, then for the sample file, the matcher says there are _no_ matches and suggests adding both ofx transactions. That's just broken, and the subject of another bug. Furthermore, if update match action is off and you sequentially import the same file, if you click 'R' because you're relatively sure you have a match in the register, the matcher displays a match score that can be acceptable (matching the already cleared split). If you always have to add incoming transactions that are a day off, then the matcher isn't doing any good at all if the user wants to maintain transaction dates in the register instead of bank posting dates. This bug is that the matcher can propose that 2 incoming transactions match the same existing register transaction, and then silently drop one of them when the matcher closes. For this file it's obvious that 2 transactions can't match one, but in a real data file where you have perhaps 100 uncleared transactions and are importing 20, it's a lot harder to tell that the matcher is setting up to delete data. I don't think the matcher should be designed to replace an already existing transaction that has cleared _and_ has an online FITID stored for it.
Oh, I definitely agree that it would be a VERY good idea to have the matcher do: a) When matching a transaction already cleared _and_ having an online FITID stored, skip the behavior of "Enable Update Match Action" even if enabled (act as if it was unchecked for that transaction). I can see no way where that would disrupt the use cases for Enable Update Match Action (which for the record are 1- In cases where the ATM fee threshold are required, not screwing up the balance as the amount downloaded from the bank is clearly the "right" one from the bank account's point of view 2- Adding in the details of the transaction from the bank when it is expected they are more complete than what's already there. Examples include ATM withdrawals and checks) b) Allow as many FITIDs as there are splits. This would greatly simplify matching many commont things, such as credit card payments that appear in two accounts. c) Even better would be to attach the FITID to the split instead of the transaction, which would mostly avoid the data loss problem your are talking about, if combined with self matching (allowing matching against a transaction part of the same download). The problem with implementing self matching is that if memory serves, the generic importer no longer creates real transactions immediately. That would make for some annoyingly boilerplate code duplication. On the other hand, the current implementation proably makes it easier to write the necessary dependency handling code (If a transaction that would have been created is changed to one that matches a current one, a transaction matching it's potential "other" split later in the matcher must be reprocessed. All this being said, you touch on two other issue: 1- "If you always have to add incoming transactions that are a day off, then the matcher isn't doing any good at all if the user wants to maintain transaction dates in the register instead of bank posting dates." The real problem is that Gnucash doesn't expose the posting date anywhere in the UI, making it very difficult indeed to maintain proper dates for checks. That's a real problem, but not an importer one. If there would be proper UI support, there would be no reason for the importer to mess with anything but date posted at all. The data model already supports it fine by the way. 2- "If you disable update match action, then for the sample file, the matcher says there are _no_ matches and suggests adding both ofx transactions. That's just broken, and the subject of another bug." Yikes, indeed, that's another bug, and a critical one at that!
I am following this very closely as I am seeing the same behavior, particularly as David R reports in comment #3. If the "R" status is supposed to be as sacred for OFX imported transactions as the "C" status is for manually reconciled transactions, then no transaction marked "R" should ever be discarded without a warning and confirmation by the user. Benoit G's comment #4 nicely fleshes out the other important considerations, except that I want to add one more issue. I often import transactions from my bank that do already have pre-existing manually entered transactions which the transaction matcher fails to recognize. In those cases, it is impossible to manually force the match because the matcher will only allow selecting a target account for that incoming transaction. Another minor UI concern may be addressable. At least in the Windows implementation, there is no visible indication in the main list which incoming transaction is being reviewed by the user for alternate matches in another window. This makes it hard to tell which ones have been reviewed and which have not. David Carlson
> I want to add one more issue. I often import transactions from my > bank that do already have pre-existing manually entered transactions which the > transaction matcher fails to recognize. Regularly? That's a bit surprising. A transaction on the same date with the same amount should have a high enough heuristic to match, if with a low confidence factor. > In those cases, it is impossible to manually force the match because the > matcher will only allow selecting a target account for that incoming > transaction. I don't understand. If you manually change from adding the transaction as new to matching an existing one, it should let you pick a transaction, not a destination account.
In one account I have three different pre-authorized debits which are not at all consistent about the date of the month on which they happen. They sometimes vary by as many as three days from the expected date, either way, especially if a weekend gets in the way. In another account I have utility bills which change in amount each month. Sometimes I find out the exact amount ahead of the game and I correct the amount, other months I get lazy and I don't. What then often happens is that when I double click on the incoming transaction, the matcher only allows me to select a target account, but not to match to any existing transactions. If there is a trick to make it let me match to existing transactions, please let me in on the secret. I have been avoiding allowing the matcher to update transactions, because I don't know exactly what will happen. When dates are off, for example, does it change the date to the bank's date or leave it as it was. I usually put the date that I pay a bill in the memo then take the bank's date for the register. Dues it overwrite the existing memo with the bank's memo? Then I would lose the paid date info and info that the bank does not know, such as whether the payee is not the real creditor, but rather an agent that collects for a lot of creditors, such as Household Bank Retail Services, for example. That reminds me of another issue. When the matcher tries to guess what account to use with a given payee, it often confuses gas stations with restaurants, but I cannot see that unless I individually check each one. Too many times I am eating gasoline or filling up my car with dinner. At least I can find and correct transactions that are assigned to Imbalance. Sorry, when I get started I just ramble on... David Carlson
I'm experiencing a version of this: If I have 2 transactions on the same date with the same amount but different descriptions/splits etc., when I import the OFX, it automatically matches both imported transactions with ONE existing transaction, leaving one unmatched. If I then accept the import, then re-import the same file, it matches the previously unmatched transaction. It also gets the one it matches in the first place wrong half the time...
This is still happening in GnuCash 2.6.3. If two imported transactions have the same date and amount, then the matcher matches both of them to one existing transaction. This gives the impression both are correctly matched (when they can't be), then leaves one transaction unmatched.
I'm bumping this since it's been two years since the last comment. It is still a bug in 2.6.12. If there are existing unmatched transactions with the same date and amount, the matcher will match all imported transactions with ONE of these, and give a result that everyrhing has matched successfully. In fact what remains is one of the transactions is matched, and the remaining ones are unmatched.
GnuCash bug tracking has moved to a new Bugzilla host. The new URL for this bug is https://bugs.gnucash.org/show_bug.cgi?id=653715. Please continue processing the bug there and please update any external references or bookmarks.