GNOME Bugzilla – Bug 105334
import multiple account OFX files with transfers between the accounts, duplicate transactions are generated
Last modified: 2018-06-29 20:27:33 UTC
Consider an OFX file describing the transactions for accounts A and B, including a transfer from A to B. This transfer will be recorded as a debit from A and a credit to B. Gnucash counts these as TWO different transactions, one has to manually disable the second one. Gnucash should be able to see somehow that the two lines are really the same transaction.
Actually, this is a pathological case. The ofx standard defines a way to specify interaccount transfers. I haven't been able to locate a sample file that actually record's it properly. If your's does and you can provide it to me, I'll implement it. But since most don't, here is why the matcher cannot find it. For matching purposes, the matcher ignores all transactions created as part of the import. Otherwise, you have two very big problems. The first is this case. If we allowed the matcher to apply match heuristics for these two transactions, it would find that BOTH are duplicates, and by default import neither. The second (similar) case is two transactions on the same day with the same memo/description. Example would include witdrawing 20$ from an atm twice on the same day, paying 8$ parking to the city twice with a credit card, writing 20$ cheques to each of your grandchildren for christmas, and having them all cash them on monday, etc... So basically, the OFX protocol planed for your specific case, and can fix it if your file includes the info. Otherwise, I simply don't know how to fix it.
Aren't the transactions associated with an account? If so, then all you need to do is "packetize" the duplicate detection as you import. You can assume that an OFX file wont have duplicates within an account, so if you packetize the duplicate detection at the account level then you don't have to worry about your second case, because the two $20 withdrawals will come together. The packetizing also helps with the former case -- you only search for duplicates ACROSS packets, not within packets. For example, the import process starts with the transactions for account A, and imports txns 1, 2, and 3. Then you get to account B, starting a new packet. At this point you add the "new" transactions from account A to the "imported" list and start reading B's txns. B has txns 4, 5, and 6, but #5 is really a duplicate of #2. At this point you merge 2/5 into a single txn.. However, let's say that 4 and 6 are both $20 ATM withdrawals. Those are NOT duplicates, because they are both within the same packet. Note that I said "merging" and not "deletion". If you have OFX transfers between two "Mutual Fund" accounts you need to be able to merge the share-price from both ends into a single transaction. However, I do believe that this algorithm will solve both problems -- assuming you have some way to packetize the import.
It wouldn't work (At least not as simply). During import, the "second" split usually isn't there yet to match against. The transactions received in account A doesn't yet point to B, and the transaction received in B doesn't yet point to A. The only way I see of fixing this for banks (all of them?) who don't send the proper transfer data is this: For each transaction, send the list of transaction_info currently stored in the matcher to the backend when doing duplicate matching. When the matcher reaches the transaction in account B, the backend could then peek at the autoselected destination account that was hopefully filled for the transaction in account A. If the destination account is "B", it could consider that transaction as a potential duplicate. A side effect of this is that it wouldn't suffer from the "transactions matching each other, and thus not being imported" problem. The first transaction to be reached by the matcher would have nothing to match against. Another side effect is that it should work for any protocol, not just ofx. Grr, I guess this make this bug assigned to me now.
I'd also like this feature. My bank (NatWest) only offers OFX, and doesn't seem to set the proper transaction information. I've read the code and Benoit's idea, and it makes sense. I'm new to the GnuCash codebase and glib, but I'll see if I can get anywhere with this. It's a pity this issue hasn't got anywhere in 5 years :-).
I have the save problem. But in my case both transaction in the OFX file have the exact save <FITID> field. couldn't this be used in the matcher to detect the duplicate AND auto assign A->B (and of course B->A). this could be done in a second pass in the matcher. first match the imported transaction to the existing ones. Then find duplication inside the imported transaction only and auto assign them. couldn't this solve both problem you've described? I can provide the file if needed
I failed to find this bug report when I created bug 751586. I can provide a sanitized example OFX import file from my bank if it helps.
GnuCash bug tracking has moved to a new Bugzilla host. The new URL for this bug is https://bugs.gnucash.org/show_bug.cgi?id=105334. Please continue processing the bug there and please update any external references or bookmarks.