Can you please also attach a (short) QIF file that can be used by the developers to reproduce this issue? Thanks.

Created attachment 98861 [details]
the 2 exmaple qif-files I used to produce the bug

Attached are the 2 example qif-files (compressed as one zip-file) I used to produce the reported bug with the memo. See also the non-ascii-characters which are dropped by the qif-importer.
The Quicken used to produce the qif-files was the German version of "Quicken Home & Buisiness 2007".

I have been researching this issue for a while, because I had the same experience. The memos don't display in GnuCash as Quicken users would expect. But this is more than a display issue, as it also affects reporting. An additional problem is that Quicken and GnuCash are somewhat incompatible in the way that memos are treated. Let me explain my thinking on this one... and I'll use the term "memo" throughout, even though the GnuCash GUI calls them "notes".

First, a review of how Quicken and the QIF format use memos:
- For non-split transactions that use a "category", one memo is allowed.
- All other non-split transactions are transfers between accounts. Two memos are allowed: one in each account register. Initially they are set equal, but you can go back and make them different if desired.
- When entering a split transaction, one memo is allowed, plus an additional memo for each split line. For each split line that references an account rather than a category, yet another memo is allowed (visible only in that account's register, and equal again by default).

Now, a review of how GnuCash uses memos:
- One "transaction" memo is allowed. This memo is viewable in every account register involved in the transaction (by turning on the "double line" viewing option).
- One additional memo is allowed for each split line. Again, these are viewable in every account register (except in "Basic Ledger" mode, which hides splits).
- GnuCash uses one more split line than Quicken/QIF, to balance the transaction. This "near" split line doesn't exist in Quicken.

Comparing these two lists, I draw the following conclusions:
1. For non-split QIF transactions that use a "category", there is only one memo, but there are three possible places to put it in GnuCash: (a) the "transaction" memo, (b) the "near" split line, and (c) the other split line. Presently the importer puts the Quicken memo in (b) and (c) but not (a). NOTE: The proposed change would put this memo in (a) ONLY.
2. For non-split QIF transactions that use an account, there are two memos in the QIF and the same three possible places to put them in Gnucash. Currently the importer drops one of the two memos - I'm not sure which. The remaining one goes in both (b) and (c). THIS IS A BUG; in the current style, one memo should be going to (b) and the other to (c). But in practice the two are usually identical anyway. NOTE: The proposed change would continue dropping one memo and put the remaining one in (a) ONLY.
3. For a QIF split transaction, there is one memo that is not part of a split line. There are two possible places to import this: (a) the GnuCash "transaction" memo, and (b) the "near" split line. Currently the importer doesn't import this memo at all! And the "near" split line's memo is duplicated from one of the other split lines. NOTE: THIS IS A BUG. This memo ought to be imported in a manner consistent with #1 and #2.
4. When importing a QIF split transaction, each QIF split line that refers to a category can be imported as a GnuCash split line, and these memos map one-to-one. No issue here. This is how the importer currently does it.
5. When importing a QIF split transaction, each QIF split line that refers to an account has two memos, so one must unfortunately be dropped. In practice this is not a big loss since the two are almost always identical. (This is how the importer currently does it.)

Now let's consider how the proposed change (see #1 and #2, above) would affect reporting. It appears to me that there are only two non-business reports that use the memos. The "Transaction Report" shows all memos, so I don't see any issue there. However the "Account Summary" shows only the split line memos, so the proposed change would drop many memos off this report. I doubt that this is an intended side effect! I don't know how big a deal this is. In any case, the report design could be changed: the transaction note could be added as a separate line on this report, or shown by default on lines where the split note is blank.

So what should be done? My opinion would be to go ahead with the proposed change, because - and I think this is really important - it matches the usual way that GnuCash users enter memos by hand. (At least that's my understanding. Correct me if I am wrong, because GnuCash is still new to me.) At the same time I would fix the bug mentioned in #3.

Sorry for the length of this comment, but I wanted to fully consider the implications... I have 15 years of data to import that would be affected by this change.

If no one is actively working on a patch, I'll volunteer.

A couple of corrections to my previous comment. I got the names of the reports wrong.  "Account Report" is the one that shows all memos. "Transaction Report" is the one that only shows split line memos and could potentially be changed (in fact, this report's design is already under discussion as bug 454834).

Created attachment 100736 [details]
QIF with fully populated memos

This testing QIF that I have been using only contains a few transactions, but the various memo fields are fully populated with unique values.

Created attachment 100783 [details] [review]
Proposed patch

This patch implements the following changes:
-For QIF non-split transactions, the memo now imports as a GnuCash "transaction" memo (seen in double-line viewing mode) rather than on the credit and debit lines.
-For QIF split transactions, the memo from the "M" line now imports as a GnuCash "transaction" memo (seen in double-line viewing mode). Previously, this was not imported at all.

As this is my first time programming in scheme, I tried to test it extensively. It appears to work as expected against all the sample QIFs posted here, and also against my own giant QIF with 15 years of personal data.

Hi Charles. Nice write-up!

First of all, for clarity, I will refer to the field by their proper names: Notes for the txn level item and Memo for the split level item (in GnuCash). Quicken appears to duplicate this structure for split txns, right? 

As to usage, I don't think you can make assumptions as to what and how users use these fields. For example, I almost never use the double-line view and thus rarely access the Notes field. There are specific instances where I do, but generally not. So in my case, having what I view as Memos from a qif file shoved into the Notes field is not good. But that's just me and my example...

Also, in my opinion, you should err on the side of preserving information. 

Case 1 above: I disagree. In this case there is only one piece of information, and there is no ambiguity as to where it belongs -- to the whole transaction, but there is also no reason to shove it into the Notes field only where is may not be visible to a lot of users. Why not maximize the information and put it in all three? 

Case 2 above: I agree this is a bug, but don't agree that the solution is to still drop one and put the other in the Notes. That doesn't really fix the bug, which is a loss of information. I think the proper fix is to actually put both of them in the proper Memo fields, and *not* the Notes field. This more accurately preserves the existing information and better matches the source, the original quicken file.

Cases 3-5: I agree with your treatment here. The txn level memo from the qif file should map to the Notes field, the category splits to the Memo field and one of the account splits to the Memo field as well. It is unfortunate that in this case some information has to be lost, but that's the way it is. 

I realize there may be no way to reconcile Case 2 with the account splits problem. 

I've reviewed your patch: you don't need to refer to bug#'s in the comments, just explain what you're doing. 

to implement getting a qif-memo into all the split Memos

(foreach
  (lambda (s)
    (xaccSplitSetMemo s qif-memo))
  (xaccTransGetSplits gnc-xtn))

untested.

let me know what your thoughts.

Regarding the bug numbers in comments, I'm happy to comment in whatever fashion you fellas see fit. However, I already asked about this in the developer's mailing list, and Christian Stimming said that bug numbers would be useful. Maybe you two could discuss it and let me know which way to go?
(see https://lists.gnucash.org/pipermail/gnucash-devel/2007-December/021751.html)

It's always hard for a new GnuCash user such as myself to judge the effects of importing things in different ways. Could someone explain what the difference is between Notes and Memos in GnuCash? I understand the differences that can be seen in the register, but what about other places? I think you said, Andrew, that Notes print on checks but Memos do not? Are there additional differences?

Andrew,

I'm curious about how you normally use the Memo field rather than the Notes field when keying in new, non-split transactions. Do you enter the same text on both db/cr lines, like the QIF importer, or do you just put the memo on one of them?

Regarding your suggestion for case 1 (to copy the QIF memo to all three spots in GnuCash), I suspect this common-denominator type of solution would partially satisfy everyone but might fully satisfy no one. The memo would definitely reach all the places that people want it to go, but it would also be duplicated (as now) or even triplicated! Maybe this is a personal preference, because I'd rather see this in the Notes field only. For these types of transactions, I never want to see the debit/credit lines in GnuCash at all. Unless you've got memos in there, the db/cr lines are pointless to display. And even then, unless the memo is relevant purely to the credit alone or the debit alone, using the Notes field would seem more appropriate.

Regarding case 2, it's true that there are two separate QIF memos that could be matched to the two db/cr lines in GnuCash, but those two memos are almost always identical. In my experience, when they are different it is totally unintentional. Assuming they're identical, I'd prefer putting the text into the Notes field. If the two memos do happen to be different, then both certainly could be kept by applying them to the db/cr line memos instead. This would be rare and I could live with that.

It sounds like we agree on cases 3-5, and that there is no dependence on user preference. We just have to figure out how to handle cases 1 & 2 so that the bug in case 3 can be patched to match.

If GnuCash needs to ask the user for a preference, then that means the druid would have to be adjusted... a much bigger change. At what point is it considered unproductive to work on improving the existing QIF importer, as opposed to working to bring the QIF importer in line with the other importers?

I should probably comment at this point that the personal preferences I've expressed above are based on how I want the register to display things, and on my desire to not maintain the same text in multiple places. If there is a significant difference between Notes and Memos outside the GnuCash register, other than for check printing, I'd like to know about it.  Thanks!

Re comment #8, in general it depends on the comment.  If you're describing WHY something is the way it is then sometimes it helps to put a bug # in there.  However, just putting the bug# in a comment by itself provides little background on a change.  So it really depends on the context and content of the comment.

Re comment #9, the main differences are that the Description and Notes fields are tied to the transaction, whereas the Memos are tied to the individual Splits.  Keep in mind that technically every transaction is a "split transaction", in that ever balanced transaction has at least two Splits.  This means that even in a basic two-split transaction you have the Description, Notes, and two memo fields (one for each of the splits).

As already stated the only major difference is that Desc and Notes are printed on checks whereas the memo fields are not.

Re comment #10, I can't answer for Andrew, but for me it very much depends on what it is I want to record.  Some things I try to record in the notes field, and some things I put in the memo field (because it's a little easier to access).  If it's something I'd ever want printed on a check I put it in the notes field.  However if it's information about the transaction itself I'd put it in the memo.

I dont think that we should copy the data to all three spots.  I think we should fill the data in the order of Description, Notes, Split memos.  Indeed, I generally think that the ONLY time we should put stuff into the individual split memos (except, perhaps, in the first split) is when we have a split QIF transaction, in which case they should get copied directly across.

I do agree that we should try not to lose data when we can, so we should strive to find a place to store the data, but I do think that when we have just a "Payee" and "Memo" from QIF, they should go into the Description and Notes fields of the transaction, and NOT into the Split memos.

A question regarding the GnuCash Description field: Why does the QIF importer put the QIF memo there when the QIF payee is unavailable? Description is allowed to be blank when entering transaction by hand in the register. If the QIF payee is blank, why not let Description be blank too?

In all cases, listen to Derek in preference to me ;).

regarding comment #10: yes it depends. I actually use the Notes field only rarely. I use a single-line auto-split register, which gives me no easy access to the Notes field. It is (I think) accessible through the invoice payment dialog, so I use it in that case, but otherwise, I generally *don't* use it. That's also why I prefer to triplicate that information in the single-split qif txn. I defer in that case, though, and given a choice between putting it in the Notes and leaving things as they are, I vote for Notes ;)

comment #12: sounds like a bug to me. If both the source and destination of the data allow no Description/Payee, then there is no reason for the importer to force that issue, IMVHO. 

oh and Memos. I put memo's in the appropriate split. The split that points to the CHecking account doesn't describe the txn like the split that points to the expense account. That split is the one that get's the memo in my world.

w.r.t comment #12: it puts it there because the Txn Description is the most visible piece of information about the transaction, and different providers of QIF supply different pieces of information.  As I said in comment #11, you should always try to put data into the description first, then the notes field, and then the split memos.  So promoting a QIF Memo to a GnuCash Description is NOT a bug, it's a feature.

Regarding comment #14: I see, so copying the QIF memo to Description is really a way of hedging your bets against inconsistent usage of the QIF format among various institutions. I take it that some frequently use blank payees but put a payee or payee-like information in the memo.

This is where importing transactions from Quicken and importing transactions from a banking institution have differences. If the importer knew that the QIF file was generated by Quicken itself, then a blank payee could properly result in a blank Description. But this is too rare a case to be worth bothering about. Accommodating inconsistent QIF content seems more valuable.

Thanks for the explanation. I was curious about that.

Derek, what is your opinion on case 2? When the two QIF memos are identical, I suspect that you would prefer to have the text go into Notes. But what if the two QIF memos are different? Would you drop one, and place the remaining one in Notes, or would you prefer to leaves Notes empty and instead use db/cr line Memos so that both memos are retained?

Could you give me an example of a QIF that has "two" memos?

Created attachment 101608 [details]
Example QIF for case 2 with differing memos

Derek, see the attached example. This QIF file represents a transfer of $100 from a savings account to a checking account. The QIF file contains two entries: one for each account. These two entries represent the two sides of the same transaction - a credit and a debit. But since each entry has its own "M" line, there are two memos. These may or may not be identical.

Ahh, I see.   In this case, yes, I would propose you drop one.  Keep the first one you see and put it in the Notes field and drop the second one.  Granted, if it's not TOO much trouble I suppose you could notice that you have two sides for the same transaction and NOT promote the Memo to the Notes field.  I think either case would be reasonable behavior.

Regarding comment 19: I've taken a closer look at the Scheme code, and it appears that keeping both QIF memos in case 2 could be a much larger change. Identifying this situation would have to be done at the stage when qif-xtn objects get merged - but at that point there is only one place available to hold a memo (in the merged qif-split). To keep both memos, I would have to modify the definition of qif-xtn or qif-split objects. Alternately, I could do a kludge by stuffing the second QIF memo in an extra, faked qif-split that would have to be recognized and processed as an exception case later on, when it came time to build a proper GnuCash transaction from it. There is already a "default-split" that may suit this purpose, but currently it's only used when processing QIF split transactions and I don't know how many parts of the code might become confused if it suddenly began appearing in non-split situations.

On the other hand, dropping one of the QIF memos is very easy because that functionality is already present in the current, unpatched QIF importer - which creates two identical Memos from the one QIF memo that remains. The previously submitted patch just uses the Notes field instead.

Bearing this in mind, I think it might be too much trouble to try to retain both memos for case 2. In my own personal experience with Quicken, the two memos only differ very rarely anyway, and even then it's unintentional on my part.

What do you think? Do we go ahead with dropping one QIF memo and placing the other in Notes?

> What do you think? Do we go ahead with dropping one QIF memo and placing the
other in Notes?

Yes, I think that should be the plan of attack.

One additional question regarding Description: When a QIF memo is promoted to become a GnuCash Description, should it also be copied to Notes or anywhere else? I ask because when the current QIF importer promotes a QIF memo to Description, it still puts a duplicate copy in the debit line memo and credit line memo.

IMHO it doesnt need to copy it to Notes or anywhere else if it's promoted to the Description.

OK, I believe we've worked through all the design decisions now. It was good to have a lengthy discussion to iron it all out. Here's a summary:

Case 1: For non-split QIF transactions that use a "category", there is only one
QIF memo, which will be copied to Notes.
Case 2: For non-split QIF transactions that use an account, there are two QIF memos. One will be dropped and the other will be copied to Notes.
Case 3: For QIF split transactions, there is one QIF memo that is not part of a split line. This will be copied to Notes.
Case 4: No changes to existing functionality.
Case 5: No changes to existing functionality.
And finally: If there is no QIF payee, any QIF memo that would become Notes will become Description instead.

Created attachment 101657 [details]
QIF with additional tests for Description

The memos now also indicate which case applies to them.

Created attachment 101658 [details] [review]
Updated patch for Description promotion

I have also updated the comments, which hopefully are to everyone's satisfaction.

Created attachment 101659 [details] [review]
Revised patch with indentation fixed

Otherwise, no changes.

I looked over your patch, and it looks good to me. It definitely cleans up my qif importing (extraneous memos floating around). 

Thanks!

committed r16758

branches/2.2 @ r16795 for GnuCash 2.2.3.
Thanks a lot!

GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=495219. Please update any external references or bookmarks.