After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 340372 - Need to run Check-and-Repair when we import datafile from 1.8
Need to run Check-and-Repair when we import datafile from 1.8
Status: RESOLVED WONTFIX
Product: GnuCash
Classification: Other
Component: Backend - XML
git-master
Other All
: Normal normal
: ---
Assigned To: Andreas Köhler
Andreas Köhler
Depends on:
Blocks:
 
 
Reported: 2006-05-02 05:54 UTC by Adam Buchbinder
Modified: 2018-06-29 21:03 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
GnuCash file to trigger the bug. (27.74 KB, text/gzip)
2006-05-02 05:56 UTC, Adam Buchbinder
Details
My obfuscation tool. (3.83 KB, application/x-perl)
2006-05-09 05:36 UTC, Adam Buchbinder
Details
My obfuscation tool. (4.10 KB, application/x-perl)
2006-05-11 03:53 UTC, Adam Buchbinder
Details

Description Adam Buchbinder 2006-05-02 05:54:15 UTC
Please describe the problem:
The 'Price Scatterplot' report fails to generate for the attached file.

Steps to reproduce:
1. Load the included GnuCash file.
2. Generate a 'Price Scatterplot' report for FUND commodity FCNTX in USD, using
'Actual Transactions'.


Actual results:
"An error occurred while running the report."

The command line reads:

;;; WARNING (wrong arguments for gnc:make-gnc-monetary:  #f #<<gnc-numeric> num:
554483 denom: 100000>)

;;; WARNING (wrong arguments for gnc:make-gnc-monetary:  #f #<<gnc-numeric> num:
554483 denom: 100000>)
In /usr/local/share/gnucash/scm/report.scm:
 412: 17* [renderer #]
In unknown file:
   ?: 18  (letrec (# #) (let* # #))
    ...
   ?: 19  (letrec (#) (gnc:html-scatter-set-title! chart report-title) ...)
In /usr/local/share/gnucash/guile-modules/gnucash/report/price-scatter.scm:
 212: 20* (if (not #) (begin # # # ...) (make-warning # #))
 214: 21  (begin (if (not #) (set! data #)) (set! data (filter # data)) ...)
 215: 22* (if (not (null? currency-accounts)) (set! data (case price-source # ...)))
 216: 23  (set! data (case price-source (# #) (# #) ...))
 218: 24* (case price-source (# #) (# #) ...)
 220: 25  [gnc:get-commodity-inst-prices (# # #) (# . 0) ...]
In /usr/local/share/gnucash/scm/commodity-utilities.scm:
 237: 26  [filter #<procedure gnc:price-is-not-zero? (elem)> ...
 239: 27*  [map-in-order #<procedure #f (a)> (# # #)]
In unknown file:
   ?: 28*  [#<procedure #f (a)> #<gw:wcp <gnc:Split*> 0x863e580>]
In /usr/local/share/gnucash/scm/commodity-utilities.scm:
 241: 29*  (let* (# # # # ...) (if # #) (list transaction-date #))
 277: 30   [list (1143781200 . 0) ...
 279: 31*   (if (not #) (begin # #) (gnc:numeric-div # # GNC-DENOM-AUTO ...))
 281: 32    (begin # #)
 282: 33*   [warn "get-commodity-inst-prices: " ... ...
 284: 34*    [gnc:commodity-numeric->string #f #]
 106: 35     [gnc:monetary->string #<<gnc-numeric> num: 554483 denom: 100000>]
In /usr/local/share/gnucash/scm/report-utilities.scm:
  49: 36     [gnc:amount->string #f (print-info # 2 2 ...)]
/usr/local/share/gnucash/scm/report-utilities.scm:49:3: In procedure
gnc:amount->string in expression (gnc:amount->string (gnc:gnc-monetary-amount
value) (gnc:commodity-print-info # #t)):
/usr/local/share/gnucash/scm/report-utilities.scm:49:3: Wrong type argument in
position 1: #f


Expected results:
The report should generate.

Does this happen every time?
Yes; this is repeatable.

Other information:
The file has been obfuscated; hence the strange names and nonsensical commodity
prices.
Comment 1 Adam Buchbinder 2006-05-02 05:56:26 UTC
Created attachment 64651 [details]
GnuCash file to trigger the bug.
Comment 2 Josh Sled 2006-05-08 00:56:47 UTC
Confirmed.
Comment 3 Chris Shoemaker 2006-05-09 02:17:41 UTC
The file contains transactions like the following one:

<gnc:transaction version="2.0.0">
  <trn:id type="guid">ae0e332467ed37d1f3e95db085014b1b</trn:id>
  <trn:num>bountiful</trn:num>
  <trn:date-posted>
    <ts:date>2006-03-31 00:00:00 -0500</ts:date>
  </trn:date-posted>
  <trn:date-entered>
    <ts:date>2006-03-17 17:51:47 -0500</ts:date>
  </trn:date-entered>
  <trn:description>antiquating</trn:description>
  <trn:splits>
    <trn:split>
      <split:id type="guid">862e2f5da8b6e2cbe360ed13388f2337</split:id>
      <split:reconciled-state>n</split:reconciled-state>
      <split:value>845954/100000</split:value>
      <split:quantity>589/100</split:quantity>
      <split:account type="guid">31cfad8379a1cc6f1a642a8a819fe488</split:account>
    </trn:split>
    <trn:split>
      <split:id type="guid">b73e0d06f880f31e43fbe7213b1ac4e8</split:id>
      <split:reconciled-state>y</split:reconciled-state>
      <split:reconcile-date>
        <ts:date>2006-04-25 00:00:00 -0400</ts:date>
      </split:reconcile-date>
      <split:value>988936/100000</split:value>
      <split:quantity>701/100</split:quantity>
      <split:account type="guid">e85bf468af821ac383978f4c4e910655</split:account>
    </trn:split>
  </trn:splits>
</gnc:transaction>

for which GnuCash can't determine the transaction currency.  I don't know what GnuCash is supposed to do in this case, but I don't see how it can make a valid transaction out if it.  It may be that the price scatterplot report is just assuming that all transactions are valid.

So, Q1: How did you end up with these transactions?  If you created them with a recent GnuCash, then there's a bug somewhere.

Q2: Can you reproduce this error with valid transactions?

Q3: If your obfuscation was automated, would you be willing to share your obfuscation tool?  I think it's something that others might find helpful when reporting bugs.



Comment 4 Chris Shoemaker 2006-05-09 02:20:07 UTC
Just to clarify:  the problem with the transaction is that no split has value == quantity, which would indicate that the transaction currency is that Split's account-commodity.
Comment 5 Adam Buchbinder 2006-05-09 05:36:19 UTC
Created attachment 65064 [details]
My obfuscation tool.
Comment 6 Adam Buchbinder 2006-05-09 05:55:56 UTC
Thanks for looking into this.

Q1: The transactions were created in gnucash 1.8.10, which I use for day-to-day work. (I now use 1.8.12.) Loading the file in gnucash 1.9.5, the error shows up.

Q2: The transactions appear to *be* valid; when I open them in 1.9.5, they look exactly the same in the register; I can save the file and reopen it; the error is still there. Does that mean it's a bug in the importer? It's supposed to transparently handle loading gnucash-1.8 data, isn't it?

Duplicating the transactions and deleting the old copies doesn't fix it; to get the transactions to be entered properly, I have to manually re-enter them.

Q3: Attached above; it uses Digest::MD5 (which I think everyone has; it doesn't *really* need it) and XML::DOM. By all means, toss it in the distribution if you'd like. I wrote it because I had issues I could only reproduce in my main working file, and I noticed that Quicken provided a similar tool for reporting bugs.
Comment 7 Derek Atkins 2006-05-10 00:22:20 UTC
Adam: can you try the Check&Repair function and see if that solves your problem?
Comment 8 Derek Atkins 2006-05-10 00:32:33 UTC
Another note:  both accounts of this txn have USD, and the transaction in question pointed out my chris isn't even balanced!  So, I suspect your randomizer is broken, and the real problem is that you ran the Log Replay (which is known in 1.8 to create broken transactions).  These transactions are fixed by Check&Repair in 1.9..
Comment 9 Adam Buchbinder 2006-05-10 01:01:04 UTC
I doubt the problem is with my obfuscator (though the obfuscator may of course be chock-full of bugs), as the problem showed up in the unobfuscated file. However, Check-and-Repair does in fact fix it--it was probably introduced one of the many times I had gnucash-1.8 die on me and had to replay the log. So I reckon this isn't really a bug, but it's plausible that you'll see this again; should the price-scatter report include a case to handle this and print a message asking the user to check and repair their gnucash file?
Comment 10 Derek Atkins 2006-05-10 01:04:54 UTC
It's not just the price scatterplot.. Lots of reports will have this problem.  And based on the fact that Check&Repair fixed it, then I suspect your obfuscator is broken.  The bug shows up because the txn has no "currency".  The fact that C&R fixed it confirms this.  C&R wouldn't work on your obfuscated file because the split values (and amounts) are "broken" (due to the obfuscator)..

Perhaps as part of the 1.8 import process we should run a C&R function.  I've renamed this bug to suggest that and left it open.
Comment 11 Adam Buchbinder 2006-05-11 03:53:54 UTC
Created attachment 65214 [details]
My obfuscation tool.

Chris Shoemaker was having some issues with this; it now fails gracefully if it can't open a dictionary; it also tries both /usr/share/dict/american-english and /usr/share/dict/words, which should cover most distributions.
Comment 12 Rolf Leggewie 2008-06-20 15:23:10 UTC
Is this still a problem that needs to be dealt with in 2.2.x?
Comment 13 Phil Longstaff 2009-05-01 13:00:50 UTC
Still a problem?
Comment 14 Phil Longstaff 2009-09-10 19:23:23 UTC
Do we care any more?
Comment 15 Derek Atkins 2009-09-11 13:00:17 UTC
I suspect we don't care per se..  Unless there's a way to detect a 1.8 data file?  I don't know if there's a way to even detect what version wrote a data file (perhaps we should add that into the data format?).

It still never hurts to run the C&R on "import"
Comment 16 Geert Janssens 2010-03-12 14:08:11 UTC
If I understand the original problem properly, there is some bug in 1.8.x's log replay functionality that creates incomplete transactions/splits. This has been fixed in the 2.0.x series, but it is possible some users still have old 1.8.x data files around (as backups for example) that may have such incomplete transactions. Such a data file can easily be fixed by running "Check & Repair" on it.

The 1.8.x series is more than 4 years old now and two major releases have been done after it. The incentive to fix conversion issues from this old release would be pretty low by now. So I don't see it useful to keep this bug open.

I don't want the information to go lost though, so I have added two sections to the GnuCash faq:

http://wiki.gnucash.org/wiki/FAQ#Q:_Can_a_new_GnuCash_release_still_read_my_old_data_file.3F and

and

http://wiki.gnucash.org/wiki/FAQ#Q:_And_the_other_way_around_.3F_Can_an_old_GnuCash_release_read_a_data_file_created_with_e_new_release.3F

These sections deal with data file compatibility between releases and suggest that "Check & Repair" should be run whenever you switch between major releases. This is not necessary in all cases, but it's easier to have users do it even when not really needed than trying to explain all the exact cases where it should be run. Most users prefer to keep it simple.

I have also created a wiki page with Adam's script as this can be useful to point users at when a user's data file is needed to fix a bug:

http://wiki.gnucash.org/wiki/ObfuscateScript
Comment 17 Derek Atkins 2010-03-12 14:15:37 UTC
Geert, we've had at least 2 users within the last month asking about loading 1.8 -> 2.2 --- So I think it is still relevant.
Comment 18 Phil Longstaff 2010-03-12 14:22:07 UTC
Sure, but I don't have a problem with documenting that after loading an old XML file, run check & repair, rather than keeping around this bug which requests that gnucash automatically run it.
Comment 19 Derek Atkins 2010-03-12 14:38:54 UTC
IMHO we should have gnucash write it's version number into a comment in the XML file, and have the parser look for that comment.. and if the file is from an older version (or doesn't have a version comment) then it should automatically run C&R.

I think automatically fixing potentially broken data files is much better than trying to teach people how to fix them when they notice something is broken...  Especially as we move forward where such errors can silently cause data loss.
Comment 20 Geert Janssens 2010-03-12 15:20:52 UTC
Ok Derek, patches are welcome ;)

But seriously, I agree that a fully automatic fix of potentially broken data files is better than teaching a user how to do it.

However, I have my doubts about the practical implementation of this.

Is a check & repair sufficient in all cases ? I vaguely remember reading the advice on the mailing lists that you should open/save a GC 1.8 file first with GC 2.0 before to open it with 2.2 due to some data file changes. (A quick check on the ML shows I'm not the only one thinking about this:https://lists.gnucash.org/pipermail/gnucash-user/2010-February/033727.html ). If this is actually true, we will have to teach the users to perform steps anyway, which are also documented in the new wiki sections.

Adding a gnucash version number to a file could still be useful though. It helps the current GC to determine how compatible it is with the datafile. Strictly speaking, versioning the data file schema itself would even be more accurate. GnuCash doesn't really care what other version of GnuCash has modified the file, it does care if it can read/modify the current schema.

I wonder though if it's not better to cast this idea into a new enhancement request rather than to add to this bug which was very specific about an update from 1.8 to 2.0.

If you agree, I'm happy to create this enhancement request, and have it refer to this old bug as a practical example of the issues it could fix.
Comment 21 John Ralls 2018-06-29 21:03:19 UTC
GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=340372. Please update any external references or bookmarks.