I am currently testing a patch to fix the lot link issue more profoundly.

The idea is that typical payments are represented again as it was in gnucash 2.4 and before. That is, when a document (bill/invoice/cn) gets paid, the payment (or part of it) is added to the document lot. Overpayments or pre-payments will be in their own lots. When these are used, again the respective payment splits are moved into the proper document lots, eliminating payment lots that may have become empty in the process.

Lot links still remain for one specific use case: when a credit note is used to offset (part of) an invoice/bill, this will be indicated by a lot link transaction. This is the only way I can link two documents in gnucash' current design philosophy.

To make it easier for the user to see what the lot link is about the memo on all of its splits now shows which documents are being linked.

This change should greatly reduce the creation of lot links. Credit notes are normally fairly uncommon.

Finally to clean up the mess that has been created in previous versions of gnucash I have also implemented a scrubbing function that will eliminate most of the lot links by moving the relevant payment split directly into the document lots where possible. After this function has run the only lot links still remaining will be those between documents as explained above.

This scrub function is called as part of the Action->Check & Repair functions whenever these functions encounter an AR/AP account.

I have published my changes to my personal account on github:
https://github.com/gjanssens/gnucash/tree/lot_links

I would very much like others to pull in this branch and abuse the new code.

The most important things to test are Action->Check & Repair on AR and AP accounts and all possible variations of payments that you are use to enter like prepayments, credit notes, partial payments, multi-document payments,... as well as deleting payments, corrections to payments,...

The program should handle them all fine (no warnings in trace file) and the invoice or owner reports should give the correct results.

I have tested many scenarios myself. I would like to see even more.

Please be sure to make a copy of your important data if you try this on your real books !

To pull my branch you can:
git checkout maint
git pull https://github.com/gjanssens/gnucash lot_links

And the build and run the branch lot_links

Oops, the commands I gave would pull the lot_links branch directly into your local maint branch. That's not what I had in mind.

Please us this instead:
git checkout maint
git pull https://github.com/gjanssens/gnucash lot_links:lot_links

Ah!  It seems to scrub the lot links I have to go via: Actions->View Lots and hit the Scrub Account button, after doing Action->Check & Repair.  I've been doing it wrong.

What were you doing before that was not working ?

Perhaps my explanation was too short.

There are two ways to scrub:
1. On the Account hierarchy tab
   - select the account to scrub (don't double-click to open the register)
   - Action->Check & Repair->Check & Repair Account

Variations on this one are to select a parent account of the account to scrub and then "Check & Repair Subaccounts", or in general "Check & Repair all accounts"

2. Second option is what you found:
   - Open the account register
   - Action->View Lots...
   - Scrub Account
There's no need to first do Action->Check & Repair as far as I'm aware of.

Looking back I see there are Check & Repair->All transactions/This transaction menu items when the account register is open. I'd have to check what these do. Maybe I should add the lot scrubber here as well.

I was just doing Action->Check & Repair from the main menu while in an open account window.  Not much happened.  I'd never seen/used the right button menu in the accounts tree window before.  Check & Repair->All seems to work fine, no more lot links.  Awesome.

I gave this a try and have a couple of comments, neither of which seems like a serious problem.

First, I discovered when I did the scrub that I have about 50 payments that weren't linked to their corresponding invoices.  I think I know why this happened, and it's really my fault, but I hadn't noticed it before because the Receivable Aging report is a bit undecided about how to handle lots linking payments and invoices.  For the purpose of calculating the amount each customer owes it ignores lots entirely and just sums all invoices and payments whether they are linked or not.  However if you tell it to ignore zero balance customers then it uses lots to determine who has a zero balance.  If a payment isn't linked to an invoice it is ignored for this purpose.  Since all the customers showed a zero balance and I hadn't told it to ignore zero balance customers, I didn't realize that the payments weren't linked to the invoices.  I fixed this with 50 some uses of Assign Payment.  I'm not sure if the report should be changed, or if so how, but it seems wrong that it mostly, but not entirely, ignores lots.

The other problem is that after the scrub of Accounts Payable there was one lot left that had only one split in it.  I tracked this down to a time last winter when I voided a payment.  One of my vendors lost a check dated Dec. 14 so I voided it and paid the invoice again on March 15.  Perhaps I shouldn't have done it this way, but it seemed like the obvious thing to do and it mostly worked ok.  This left things (before the scrub) with these three lots:

  Lot 1: Voided check
         Internal link dated March 15
         Internal link dated Dec. 14

  Lot 2: New check
         Internal link dated March 15

  Lot 3: Internal link dated Dec. 14
         Bill

GnuCash seems to figure out that the new payment goes with the bill even though it's not the normal way to link payments to bills.  It has to go through all three lots to get from the payment to the bill. 

All three lots show up in the lot viewer, but lot 1 only shows the two link splits.  The viewer seems to ignore the split for the voided check.

After the scrub there is a new lot with a new GUID containing the reissued check and the bill.  Old lots 2 and 3 are gone entirely.  The old lot 1 is still there (with the same GUID) and contains only the voided check.  It probably should have been deleted by the scrub although it doesn't hurt anything.

(In reply to comment #6)
> I was just doing Action->Check & Repair from the main menu while in an open
> account window.  Not much happened.  I'd never seen/used the right button menu
> in the accounts tree window before.  Check & Repair->All seems to work fine, no
> more lot links.  Awesome.

Thanks for your feedback. Great it works for you as well.

BTW, I didn't mean to refer to a right-click menu. I was just being lazy and didn't put the full menu entries. All menu options I referred to are under
Action->Check & Repair

(In reply to comment #7)
> I gave this a try and have a couple of comments, neither of which seems like a
> serious problem.
> 
> First, I discovered when I did the scrub that I have about 50 payments that
> weren't linked to their corresponding invoices.  I think I know why this
> happened, and it's really my fault, but I hadn't noticed it before because the
> Receivable Aging report is a bit undecided about how to handle lots linking
> payments and invoices.  For the purpose of calculating the amount each customer
> owes it ignores lots entirely and just sums all invoices and payments whether
> they are linked or not.  However if you tell it to ignore zero balance
> customers then it uses lots to determine who has a zero balance.  If a payment
> isn't linked to an invoice it is ignored for this purpose.  Since all the
> customers showed a zero balance and I hadn't told it to ignore zero balance
> customers, I didn't realize that the payments weren't linked to the invoices. 
> I fixed this with 50 some uses of Assign Payment.  I'm not sure if the report
> should be changed, or if so how, but it seems wrong that it mostly, but not
> entirely, ignores lots.

You do provide some interesting corner cases to test the code against :)

I would also consider a bug in the aging report and unrelated to the issue of lot link scrubbing. I would have to think about what should be the best way to handle this. But let's keep that discussion for a separate bug. I have create bug 735907 for this purpose.

(In reply to comment #7)
> The other problem is that after the scrub of Accounts Payable there was one lot
> left that had only one split in it.  I tracked this down to a time last winter
> when I voided a payment.  One of my vendors lost a check dated Dec. 14 so I
> voided it and paid the invoice again on March 15.  Perhaps I shouldn't have
> done it this way, but it seemed like the obvious thing to do and it mostly
> worked ok.

Another very interesting corner case. I'm fairly sure Derek never intended for a business-generated transaction to be voided. And likewise it never occurred to me a payment could have been voided.

>  This left things (before the scrub) with these three lots:
> 
>   Lot 1: Voided check
>          Internal link dated March 15
>          Internal link dated Dec. 14
> 
>   Lot 2: New check
>          Internal link dated March 15
> 
>   Lot 3: Internal link dated Dec. 14
>          Bill
> 
> GnuCash seems to figure out that the new payment goes with the bill even though
> it's not the normal way to link payments to bills.  It has to go through all
> three lots to get from the payment to the bill. 
> 
> All three lots show up in the lot viewer, but lot 1 only shows the two link
> splits.  The viewer seems to ignore the split for the voided check.
> 
> After the scrub there is a new lot with a new GUID containing the reissued
> check and the bill.  Old lots 2 and 3 are gone entirely.  The old lot 1 is
> still there (with the same GUID) and contains only the voided check.  It
> probably should have been deleted by the scrub although it doesn't hurt
> anything.
The scrub function is focussed purely on eliminating lot links. And I'm pleased to read it does so satisfactorily despite your unusual lot setup. Apart from that it's very conservative in keeping everything as it was. That is: all unused payments are left in their respective lots. Since the voided check doesn't pay any bill, it will remain in its lot. You can easily remove it yourself from the lot manually via the Lot Viewer if you don't want to keep the lot around.

However it bothers me slightly that a new lot was created for the bill and the reissued check. That was not supposed to happen. Payment should be moved from their respective lots *TO* the bill's lot. So in principle payment lots can disappear when payments are "consumed" to pay bills, but the bill's lot should only be updated. I'll investigate if I can reproduce this.

(In reply to comment #10)
> You can easily remove it
> yourself from the lot manually via the Lot Viewer if you don't want to keep the
> lot around.
Ehr, not *that* easily... but still possible.
To clean this up manually, you'd have to
- unvoid the transaction
- take the split out of the lot via the lot viewer (which will automatically delete the lot)
- void the transaction again.

> However it bothers me slightly that a new lot was created for the bill and the
> reissued check. That was not supposed to happen. Payment should be moved from
> their respective lots *TO* the bill's lot. So in principle payment lots can
> disappear when payments are "consumed" to pay bills, but the bill's lot should
> only be updated. I'll investigate if I can reproduce this.

And I can't reproduce this. With my simple setup (one bill, one payment (voided), one later payment), the bill lot is not replaced. Only the lot which held the reissued payment is deleted. The lot for the voided payment remains as well, and will only hold the voided payment (which is hidden in the lot viewer).

> I would also consider a bug in the aging report and unrelated to the issue of
> lot link scrubbing. I would have to think about what should be the best way to
> handle this. But let's keep that discussion for a separate bug. I have create
> bug 735907 for this purpose.

As you may have noticed I checked in a change to consider lots when calculating the balance in the owner tree code.  This seemed like a more obvious bug to me (and in code I last changed anyway).  I haven't done anything about the aging report.

> > However it bothers me slightly that a new lot was created for the bill and the
> > reissued check. That was not supposed to happen. Payment should be moved from
> > their respective lots *TO* the bill's lot. So in principle payment lots can
> > disappear when payments are "consumed" to pay bills, but the bill's lot should
> > only be updated. I'll investigate if I can reproduce this.
>
> And I can't reproduce this. With my simple setup (one bill, one payment
> (voided), one later payment), the bill lot is not replaced. Only the lot which
> held the reissued payment is deleted. The lot for the voided payment remains as
> well, and will only hold the voided payment (which is hidden in the lot
> viewer).

I can believe that voiding a payment is not something that anyone considered when writing this code.  Perhaps I should have unposted the invoice first.  

It was some time ago that I did this and I may not be remembering all the steps I went through.  It's too bad you can't reproduce it.  If I can I'll let you know.

(In reply to comment #12)
> As you may have noticed I checked in a change to consider lots when calculating
> the balance in the owner tree code.  This seemed like a more obvious bug to me
> (and in code I last changed anyway).  I haven't done anything about the aging
> report.
> 
I did see that commit and am still pondering if this is really a bug or not. I'll comment on the commit itself instead of here.

> > And I can't reproduce this. With my simple setup (one bill, one payment
> > (voided), one later payment), the bill lot is not replaced. Only the lot which
> > held the reissued payment is deleted. The lot for the voided payment remains as
> > well, and will only hold the voided payment (which is hidden in the lot
> > viewer).
> 
> I can believe that voiding a payment is not something that anyone considered
> when writing this code.  Perhaps I should have unposted the invoice first.  
> 
Well you did create an interesting test case by voiding a payment and all things considered I think the business code handles it fairly well. Except maybe for the lot remaining with the invisible split in the lot viewer. But that's mostly a visual glitch. It doesn't seem to have any impact on the accounting aspects.

> It was some time ago that I did this and I may not be remembering all the steps
> I went through.  It's too bad you can't reproduce it.  If I can I'll let you
> know.
Ok. Though even if the lot gets replaced the code seems to do The Right Thing. So it's only a minor issue. If you can reproduce I'll look into it just to be sure.

Anyway, I have now pushed the code to maint so it will appear in 2.6.4 at the end of the month.

(In reply to comment #5)
> Looking back I see there are Check & Repair->All transactions/This transaction
> menu items when the account register is open. I'd have to check what these do.
> Maybe I should add the lot scrubber here as well.

I have added (business) lot scrubbing to these menu items as well.

A closing note: It's been a while since the initial introduction of lot links and the issues the first implementation had.

I believe most of these issues have been dealt with.
- Lot links will only be used when offsetting documents (a bill with a credit note for example). They are no longer used for every payment. As a sidenote, I have come to think of them as "payments for multiple documents without a typical payment split".
- I have continued to improve the lot link scrubbing code until recently. The most recent improvements have been released with gnucash 2.6.15. I believe it's current state is as good as it will get.

So I think it's time to close this bug. If any issue with lot scrubbing is still found, feel free to report it in a new bug.

GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=734921. Please update any external references or bookmarks.