After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 114724 - QIF importer drops split details for transfers with identical amount
QIF importer drops split details for transfers with identical amount
Status: VERIFIED FIXED
Product: GnuCash
Classification: Other
Component: Import - QIF
unspecified
Other Linux
: Normal normal
: ---
Assigned To: Derek Atkins
Derek Atkins
Depends on:
Blocks: backport
 
 
Reported: 2003-06-08 18:24 UTC by Andy Fyfe
Modified: 2018-06-29 20:33 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
sample qif file demonstrating problem (from Mac Quicken 2003) (4.65 KB, application/octet-stream)
2003-06-08 18:25 UTC, Andy Fyfe
  Details
the gnucash file obtained after loading the previously attached qif file (5.38 KB, application/octet-stream)
2003-06-09 00:13 UTC, Andy Fyfe
  Details
qif file with a second transaction, differing transfer amounts (4.84 KB, application/octet-stream)
2003-06-10 04:05 UTC, Andy Fyfe
  Details
resulting gnucash xml file, after loading 2-transaction qif file (8.00 KB, application/octet-stream)
2003-06-10 04:07 UTC, Andy Fyfe
  Details
Additional QIF file to demonstrate bug as seen in 2.2.1 (609 bytes, text/plain)
2007-12-01 04:26 UTC, Charles Day
  Details
Additional QIF file to demonstrate bug as seen in 2.2.2 (453 bytes, text/plain)
2008-01-02 07:25 UTC, Charles Day
  Details
Proposed patch for testing & comment (2.74 KB, patch)
2008-01-03 02:29 UTC, Charles Day
none Details | Review

Description Andy Fyfe 2003-06-08 18:24:42 UTC
When I have a transaction that has a multipart split, and the split
involves transfers to two different account, each for the same $ amount,
gnucash keeps only 1 on import.  If the $ amount differs, both will be taken.
Comment 1 Andy Fyfe 2003-06-08 18:25:50 UTC
Created attachment 17321 [details]
sample qif file demonstrating problem (from Mac Quicken 2003)
Comment 2 Derek Atkins 2003-06-08 23:26:59 UTC
It looks like this is the transaction in question, correct?

D6/8/03
PPay
M
T100.00
A
A
A
A
A
A
LSalary
SSalary
E
$300.00
S[401k]
E
$-100.00
S[ESPP]
E
$-100.00
^

If I just load this transaction on its own then the parser loads it
just fine and it gets Split properly, so there clearly isn't anything
wrong with the parser.  But there is certainly something else going on.

What is the "source account" of this data file?
Comment 3 Andy Fyfe 2003-06-09 00:13:33 UTC
Created attachment 17334 [details]
the gnucash file obtained after loading the previously attached qif file
Comment 4 Andy Fyfe 2003-06-09 00:14:50 UTC
I observed this behaviour when loading my actual qif file into
gnucash.  To generate a simpler example, I had quicken create a new
file, to which I added three accounts -- checking, a bank account;
401k, a portfolio account, and ESPP, a cash account.  I created a
single transaction, a mock pay cheque for $300, of which $100 went to
the 401k account and $100 to the ESPP account, leaving $100 in the
checking account.  When loaded into gnucash, I get $200 in checking,
$100 in 401k and $0 in ESPP.  Though this example doesn't demonstate
it, based on my full qif file, this behaviour only happened when the
transfer amounts were identical; when they differed, the split was
entered as expected.
Comment 5 Derek Atkins 2003-06-09 20:21:02 UTC
Very strange.  The fact that different amounts are treated differently
is just..... STRANGE!
Comment 6 Andy Fyfe 2003-06-10 04:04:03 UTC
I have gone back to the original quicken file and added a second
transaction, again for $300 in salary, $100 to the 401k but only $50
to the ESPP account.  All parts are added, though the ESPP shows up
under a split transaction in the checking account, while the 401k part
as a separate line item, which is a bit strange...
Comment 7 Andy Fyfe 2003-06-10 04:05:55 UTC
Created attachment 17373 [details]
qif file with a second transaction, differing transfer amounts
Comment 8 Andy Fyfe 2003-06-10 04:07:47 UTC
Created attachment 17374 [details]
resulting gnucash xml file, after loading 2-transaction qif file
Comment 9 Andy Fyfe 2003-06-10 04:20:28 UTC
I made another discovery.  If I rename the "401k" account "Retirement"
or "My 401k", the split is handled correctly, but not if I rename it
"401k account".

So, something to do with the name starting with digits?  It's a bit
odd because it's the ESPP split line that gets dropped.  And no matter
the name, the transfer to that account does not show up as part of the
split it seems, but as a separate line in the checking register --
maybe it has something to do with the account being a portfolio?
Comment 10 Derek Atkins 2003-06-10 22:33:18 UTC
I'll take ownership, although I can't guarantee anything being fixed.
Comment 11 Andy Fyfe 2003-06-14 17:05:52 UTC
I tried changing all the names of my accounts in my original quicken
file, so that none started with a digit, and there were no non
alphnumeric characters.

Unfortunately, the problem remains.  In practice, a pay cheque would
have on the order of a dozen or so split entries, and in the cases
where  the 401k deduction matched the espp deduction, the espp
deducution was lost.

The previous suggestion that an account name that starts with digits
is related is probably a red herring, though the specifics of the
account name may still be involved somehow.
Comment 12 Derek Atkins 2003-06-30 02:56:43 UTC
Having explored, the problem appears to be the order in which the
transactions are listed in the file.  By changing the name of the
account you probably changed the order in the file such that the split
transaction occurred earlier in the file..  I think the problem is the
import not properly merging transactions.
Comment 13 Derek Atkins 2003-06-30 03:07:34 UTC
fixed summary
Comment 14 Derek Atkins 2003-06-30 03:34:28 UTC
Another thing I noticed... If you change your 401k account from an
Investment to a Bank account type it seems to import correctly...
I'll also note that the problem here is not that the 401k split has
the same value as the ESPP split -- if I change both the 401k split
and the '401k account transaction' to be 100.01 instead of 100.00, it
still imports incorrectly.  So the problem is not matching the ESPP value.
Comment 15 Christian Stimming 2006-08-04 09:11:15 UTC
Does this issue also occur in the 2.0.x versions? Development on 1.8.x has stopped, so please upgrade to 2.0.x (most current is 2.0.1) and see whether this problem still occurs.
Comment 16 Derek Atkins 2006-08-04 13:30:50 UTC
I suspect this bug still exists in 2.0, but I haven't tested it.
Comment 17 Rolf Leggewie 2007-04-26 13:26:40 UTC
Any news on this one?
Comment 18 Charles Day 2007-12-01 04:26:09 UTC
Created attachment 99947 [details]
Additional QIF file to demonstrate bug as seen in 2.2.1
Comment 19 Charles Day 2007-12-01 04:38:12 UTC
The additional QIF file that I have attached demonstrates the bug in 2.2.1:
1. From a savings account, withdraw $300 cash and pay a $1.50 ATM fee (a split transaction)
2. From a checking account, on the same day, withdraw $300 cash
3. The split transaction is imported perfectly.
4. The bug: the second transaction, a withdrawal from checking, is instead imported as a second withdrawal from savings. 

NOTE: if the order of the last two transactions in the file are reversed, the import works correctly.
Comment 20 Charles Day 2008-01-02 07:21:38 UTC
Please ignore my two previous comments, because the example I gave is now known to be a separate bug.  Filed as bug #506810.
Comment 21 Charles Day 2008-01-02 07:25:49 UTC
Created attachment 101975 [details]
Additional QIF file to demonstrate bug as seen in 2.2.2

This test file was provided to the mailing list by William Hamblen.
Comment 22 Charles Day 2008-01-03 02:29:20 UTC
Created attachment 102026 [details] [review]
Proposed patch for testing & comment

The patch is that the merging code that was at the bottom has been moved to the top, allowing all split transactions to have priority over those with actions and securities.
Comment 23 Charles Day 2008-01-03 03:07:12 UTC
I want to try to describe how the QIF importer is currently working, how the bug occurs, and what the patch does...

In QIF files, transfers between accounts show up as two separate entries: one in the "from" account, and one in the "to" account. The QIF importer has to look for these two "splits" and match them together to avoid creating duplicate transactions in GnuCash. Once a match is found, the QIF importer must decide which split to keep and which to discard.

The way it currently operates, if a transfer involves an investment account then the split in the investment account always gets priority. The other split of the matching pair gets discarded. If that other half happens to be part of a QIF split transaction (one with "S" lines), then that means that the QIF importer is discarding one of those split lines, thereby mangling the QIF split transaction. This is the bug.

What the patch I wrote does is to change the rules, so that if a QIF split transaction is involved in a match then it always has priority. The other half of the match will always be the half that gets discarded, even if it is from an investment account.

I tested this patch on the most recent QIF file that I attached (from William Hamblen) and it fixed the problem. I also tested this patch on 15 years of my own personal data, and it didn't seem to break anything else.

However, seeing as I have changed what would seem to be a rather sensitive spot in the code, I'd appreciate any/all feedback, test results, or possible unintended consequences. Thanks!
Comment 24 William Hamblen 2008-01-04 02:10:13 UTC
I've tested this on my real data (17 years worth of Quicken data exported from Quicken Basic 2004 to one gigantic QIF) and everything looks perfect.  I've checked share and dollar balances for all non-zero Quicken accounts (but not categories) and the imported GnuCash accounts match in every case.

At least for my data, this patch is a tremendous improvement.  Thanks Charles!
Comment 25 Derek Atkins 2008-01-06 16:02:26 UTC
Hi, for the future, it helps if you run a diff from the top-level of the tree so we don't need to figure out where in the tree the patch needs to apply.

I've applied this to trunk as r16828.
Awaiting backport.

Thanks!


Note that I still have a concern that if there's an investment account transaction involved you might lose the investment information, like shares/price.
Comment 26 Andreas Köhler 2008-01-06 16:28:57 UTC
branches/2.2 @ r16830 for GnuCash 2.2.3.
Thanks a lot!
Comment 27 Charles Day 2008-01-07 00:49:37 UTC
Derek, when I wrote the patch I was having the same concern, i.e. can investment information be lost? But looking at all the special cases that were set out, I still couldn't think of an example where a QIF split transaction that includes a QIF account transfer would involve a security. It worked for two of us, but if it doesn't work for others then maybe some code to copy the investment stuff could be tacked on. I guess we'll have to wait and see.

About the diff, I assume that by the top of the tree you mean one level above "src"?
Comment 28 Derek Atkins 2008-01-07 04:10:45 UTC
Yeah, I mean, effectively, "svn diff" from the top level (trunk, or branches/2.2) directory.
Comment 29 John Ralls 2018-06-29 20:33:23 UTC
GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=114724. Please update any external references or bookmarks.