GNOME Bugzilla – Bug 611936
Sqlite3: assertion `gnc_numeric_check(amt) == GNC_ERROR_OK' failed
Last modified: 2018-06-29 22:36:14 UTC
This is with a clean build starting from svn r18833. I saved an xml book to sqlite3 format. This seems to work ok. I can add splits. Then I restart GnuCash. My sqlite3 data file loads, but it doesn't contain any amounts. All other information is still there (transactions, descriptions, split details, but no amounts. My gnucash.trace file is filled with lines like: * 19:43:24 CRIT <gnc.engine> xaccSplitSetValue: assertion `gnc_numeric_check(amt) == GNC_ERROR_OK' failed * 19:43:24 CRIT <gnc.engine> xaccSplitSetAmount: assertion `gnc_numeric_check(amt) == GNC_ERROR_OK' failed Let me know if you need more information.
Oh yes, if I open the file with an sqlite browser, the amounts and values are still there, so it seems the problem is somewhere in the file loading code.
Created attachment 155476 [details] Testfile exhibiting the problem The attached test file is created with GnuCash 2.3.10 on Mandriva 2010. Steps: - Open GnuCash - Create new file with standard set of accounts - Enter one transaction - Save as sqlite3 file - Restart GC => The file is reopened, the transaction is there, but the amounts are not. I have run this test with GC 2.3.10 but I also had this problem with r18833 and currently r18861. I have these dbi libraries installed: libdbi-debug-0.8.3-3mdv2010.0 libdbi0-0.8.3-3mdv2010.0 libdbi-devel-0.8.3-3mdv2010.0 libdbi-drivers-dbd-pgsql-0.8.3-6mdv2010.0 libdbi-drivers-dbd-sqlite3-0.8.3-6mdv2010.0 libdbi-drivers-debug-0.8.3-6mdv2010.0 libdbi-drivers-dbd-mysql-0.8.3-6mdv2010.0
Following your instructions, I had no problem. Can you set a breakpoint in routine load_numeric and try debugging? The routine should get integer values for <x>_num and <x>_denom (where <x> represents an overall column name e.g. amount or value) and creates a gnc_numeric. It then either calls a routine directly or sets a gobject property. For a split, gobject properties are set in routine gnc_split_set_property. If you can trace through and find where it's going wrong, I might be able to fix it.
It seems to go wrong quite early. I set a breakpoint in load_numeric and traced into gnc_sql_row_get_value_at_col_name. The second parameter (buf) is "quantity_num" at this point. In this routine, - type gets set to 1 (means DBI_TYPE_INT - ok) - attrs get set to 32 (means DBI_INTEGER_SIZE8 - seems ok) At this point I have been running some print gdb commands. There are 12 columns in the row (correct). The names of the columns match the split table's columns. I can get all the correct values from the text columns (*guid, reconcile_state) But all the bigint fields are 0. gdb command used: print (const char*)dbi_result_get_longlong (dbi_row->result, "quantity_num") and variants thereof to get "quantity_denom", "value_num" and "value_denom". I also tried replacing longlong with long or int, but the result is unchanged. Any clue how to debug this further ?
If you go back before r18833, do you still have this problem? If you do the same debugging, do your print commands return the correct values? As far as I know, nothing has changed recently around gnc_sql_row_get_value_at_col_name(). However, I do know that there are some issues in libdbi 0.8.3 related to column size and endianness (maybe other things). Is your Mandriva installation new? New packages? What else has changed?
I am running this Mandriva installation for a couple of months now. But I never ran tests with sqlite3 before. I always used postgres so far. So the bug can be old. libdbi 0.8.3 is what comes by default with Mandriva 2010. I didn't install anything fancy. I see that Fedora 12 has 0.8.3 as well. Time is limited now, but I'll build gnucash on Fedora later and see if it has got the same issues. For me personally, the sqlite3 support is not overly important. I'll probably switch to postgres once 2.4 is out. But if sqlite3 is to become the future default file format, we'll have to find the cause of this. I'll see what more infomation I can gather.
I built r18881 on Fedora 12 for comparison. The values are still empty. So the issue is not Mandriva specific. Fedora ships these versions: libdbi-drivers-0.8.3-5.fc12.i686 libdbi-devel-0.8.3-3.fc12.i686 libdbi-dbd-sqlite-0.8.3-5.fc12.i686 libdbi-dbd-pgsql-0.8.3-5.fc12.i686 libdbi-dbd-mysql-0.8.3-5.fc12.i686 libdbi-0.8.3-3.fc12.i686 So both distros are stuck with libdbi 0.8.3. In reply to comment 6: > If you go back before r18833, do you still have this problem? I created the test db in the 2.3.10 release, which is r18662. So far, I didn't go back further than that. I have also tried to start gnucash in different languages (prepending LANGUAGE=xyz to the startup command) to test if locale has any impact. It doesn't apparently. The values are missing regardless of the language chosen.
32bit? 64bit? Did this used to work but stopped working?
32bit on both systems. I don't know if it used to work before. I just started doing some tests on the sqlite3 driver. What libdbi version do you use ?
I thought it was 0.8.3 but it's 0.8.2-3. Libdbi's been at 0.8.3 for a long time. There have been lots of bug fixed and changes since it was released, but no new version has been released.
I am apparently not the only one having this problem: https://lists.gnucash.org/pipermail/gnucash-devel/2010-June/028720.html (in the lower part of the list message the user confirms having the same issue on Fedora 13).
I just tested on OSX with r19392 and libdbi-0.8.3, libdbi-drivers-0.8.3.1, and sqlite-3.6.15. It works fine, both from a freshly saved db and one almost a year old. I use Debian for Linux, and it's still on libdbi-0.8.2.3, so I doubt that building a gnucash-svn there would be much help. Perhaps you could try building libdbi, libdbi-drivers, and sqlite3 from source and linking gnucash against them.
I get the same error and output on Fedora 12 with libdbi-dbd-sqlite-0.8.3-5.fc12.i686 and libdbi-dbd-mysql-0.8.3-5.fc12.i686
I built libdbi-0.8.3 and libdbi-drivers-0.8.3-1 from source on Debian and it runs just fine. I can save and retrieve freshly saved sqlite3 files and read old ones -- even ones written from OSX.
I've removed the sqlite rpm -e --nodeps sqlite yum remove sqlite-devel yum remove libdbi yum remove libdbi-dbd-mysql yum remove libdbi-dbd-sqlite Dsqlite-3.7.0.1ownloaded sources: sqlite-3.7.0.1 libdbi-0.8.3 libdbi-drivers-0.8.3-1 Built all and installed to /usr/local/... On saving I get the error dialog with: "The server at URL sqlite3:///PATH/sqlite.gnucash experienced an error or encountered bad or corrupt data." On subsequent re-loading of that file all transactions are blank with zero balance. However after doing "make debug" in libdbi-0.8.3, subsequent re-loading of the same file shows transactions as they should be. It appears then that the issue is with libdbi? The save error still exists though when saving data from an xml to sqlite file. Not sure where to go from here but I guess we can say it's not GnuCash that's at fault. Thanks John for suggesting sources.
See also the gnucash-devel list thread on this topic starting here: https://lists.gnucash.org/pipermail/gnucash-devel/2010-August/029141.html From the reports there (some of which are included here in the comments already) the conclusion would be this is a problem in libdbi, and not in GnuCash. In short: * The problem currently only appears in Fedora (12 and 13 are tested), 32-bit. * Building libdbi (or libdbi-drivers) on that platform with "make" results in binaries that have this bug, building with "make debug" results in binaries to work fine. It doesn't matter if the source for the build is the original tarball from the libdbi homepage or the Fedora source rpms for libdbi and libdbi-drivers.
Geert, You said in comment 2 that it reproduced also on Mandriva 2010. That it works in debug but not in release (of libdbi) suggests that there might be some optimization problem going on. Configure has some interesting settings that might affect the build depending on whether $GCC is defined in the environment. In particular, -ffast-math is used for release but not for debug; that is known to have some interesting side effects, especially for floating point computation. It something that is also likely to be sensitive to compiler version -- something that both Debian and Apple are quite conservative about (Apple is on 4.2.1, Debian on 4.4.3). Perhaps you and/or Mike could try overriding some of those flags. (Configure puts your environment CFLAGS at the end, so if you set, e.g., -fno-fast-math it will override the -ffast-math set by configure.
As another test I reverted back to a GnuCash version in mid May (just before Rick Rankin reported a problem), with the results: make in libdbi (SRPM) still same result with no data; make debug all data OK; However in neither case do I get a pop-up dialog with an error message when saving to either mysql or sqlite. Not sure if this is now two separate issues or are they related? svn r19202:19419M on Fedora 12 32bit
(In reply to comment #17) > Geert, > You said in comment 2 that it reproduced also on Mandriva 2010. > Indeed I did, and that's the platform I hit the bug on first. Sorry for omitting this. In my mind Mandriva is a product derived from Fedora so I tend to only talk about Fedora sometimes. > That it works in debug but not in release (of libdbi) suggests that there might > be some optimization problem going on. Configure has some interesting settings > that might affect the build depending on whether $GCC is defined in the > environment. In particular, -ffast-math is used for release but not for debug; > that is known to have some interesting side effects, especially for floating > point computation. It something that is also likely to be sensitive to compiler > version -- something that both Debian and Apple are quite conservative about > (Apple is on 4.2.1, Debian on 4.4.3). > > Perhaps you and/or Mike could try overriding some of those flags. (Configure > puts your environment CFLAGS at the end, so if you set, e.g., -fno-fast-math it > will override the -ffast-math set by configure. I have redone the srpm tests where I changed the spec files to execute make CFLAGS="-O20 -fno-fast-math -D_REENTRANT -fsigned-char -std=gnu99" ${smp_flags} instead of simply make ${smp_flags} The net result is that the bug disappears. Obviously the fast-math optimization is the cause of trouble here.
(In reply to comment #18) > As another test I reverted back to a GnuCash version in mid May (just before > Rick Rankin reported a problem), with the results: > make in libdbi (SRPM) still same result with no data; > make debug all data OK; > However in neither case do I get a pop-up dialog with an error message when > saving to either mysql or sqlite. > > Not sure if this is now two separate issues or are they related? > > svn r19202:19419M on Fedora 12 32bit I think the pop-error went away because a got a new SVN snapshot. It seems that over the months some errors have crept in, likely my fault. Original error still stands though.
Is this pop-up error dialog what you reported in bug 626451 ?
(In reply to comment #21) > Is this pop-up error dialog what you reported in bug 626451 ? Ah! yes it is. I'll add a note there to admit blame and allow the bug to be closed.
Is this problem now that the sql save fails silently instead of building dbi with fast-math causes trouble on Fedora11 and Mandriva10 systems (which we can't do anything about)? If it is, would someone who can reproduce the error please try with a current svn build of Gnucash? I've reworked the error handling in the sql backend in response to bug 632166, and I think that the error should now result in a (rather terse, unfortunately) error dialog and a failure to write the sql database.
I should have noted here that this was reported as https://bugzilla.redhat.com/show_bug.cgi?id=629964 and has now been resolved for Fedora, at least as far as fc12,13 and 14 are concerned.
Yes, I can confirm this is fixed in Fedora 13. Closing this as NOTGNOME now since the bug was not in our codebase.
*** Bug 632346 has been marked as a duplicate of this bug. ***
Sorry, bug 632346 is only partly a duplicate. John has removed the duplicate status of that bug, but that is not reflected here.
This is also happening in Ubuntu 10.10 i386. Bug "gnucash 2.3 with sqlite retrieves all numbers as zero" reported to Ubuntu bugtracker https://bugs.launchpad.net/ubuntu/+source/libdbi-drivers/+bug/673307
RedHat has apparently run into similar issues: https://bugzilla.redhat.com/show_bug.cgi?id=452591 https://bugzilla.redhat.com/show_bug.cgi?id=474260 Interestingly enough, that bug states gcc 4.3.2 miscompiles SQLite with optimization above -O1. It's *not* the -ffast-math problem; it's something else. The problem does not appear unless function inlining is enabled.
*** Bug 638414 has been marked as a duplicate of this bug. ***
Until libdbi is fixed, expect problems with: * longlong * ulonglong * double * datetime * datetime_tz as tested in test_dbi (part of the libdbd source distribution). Comparison of Ubuntu 10.10 and FreeBSD results: jeff@fx:~$ diff -u test_dbi.output* --- test_dbi.output 2011-01-01 10:42:03.707828037 -0800 +++ test_dbi.output.freebsd 2011-01-01 10:43:55.856451151 -0800 @@ -2,15 +2,15 @@ Driver information: ------------------- Name: sqlite3 - Filename: /usr/lib/dbd/libdbdsqlite3.so + Filename: /usr/local/lib/dbd/libdbdsqlite3.so Desc: SQLite3 database support (using libsqlite3) Maintainer: Markus Hoenicka <mhoenicka@users.sourceforge.net> URL: http://libdbi-drivers.sourceforge.net Version: dbd_sqlite3 v0.8.3-1 - Compiled: Aug 7 2010 + Compiled: Jan 1 2011 Successfully connected! - Using database engine version 30702 (numeric) and 3.7.2 (string) + Using database engine version 30704 (numeric) and 3.7.4 (string) Test 1: List databases: Got result, try to access rows @@ -47,10 +47,10 @@ the_ushort: in:32767 out:32767<< the_long: in:-2147483648 out:-2147483648<< the_ulong: in:2147483647 out:2147483647<< -the_longlong: in:-9223372036854775807 out:0<< -the_ulonglong: in:9223372036854775807 out:0<< +the_longlong: in:-9223372036854775807 out:-9223372036854775807<< +the_ulonglong: in:9223372036854775807 out:9223372036854775807<< the_float: in:3.402823466E+38 out:3.402823e+38<< -the_double: in:1.7976931348623157E+307 out:0.000000e+00<< +the_double: in:1.7976931348623157E+307 out:1.797693e+307<< the_driver_string: in:'Can 'we' "quote" this properly?' out:'Can 'we' "quote" this properly?'<< the_quoted_string: in:'Can 'we' "quote" this properly?' out:'Can 'we' "quote" this properly?'<< the_quoted_string_copy: in:'Can 'we' "quote" this properly?' out:'Can 'we' "quote" this properly?'<< @@ -58,11 +58,11 @@ the_escaped_string_copy: in:'Can 'we' "escape" this properly?' out:'Can 'we' "escape" this properly?'<< the_empty_string: out:''<< the_null_string: out:'(null)' -the_datetime: in:'2001-12-31 23:59:59' out:2001-12-31 0:0:0 -the_datetime_tz: in:'2001-12-31 23:59:59 -10:00' out:2001-12-31 0:0:0 +the_datetime: in:'2001-12-31 23:59:59' out:2001-12-31 23:59:59 +the_datetime_tz: in:'2001-12-31 23:59:59 -10:00' out:2002-1-1 9:59:59 the_date: in:'2001-12-31' out:2001-12-31 -the_time: in:'23:59:59' out:0:0:0 -the_time_tz: in:'23:59:59-10:00' out:0:0:0 +the_time: in:'23:59:59' out:23:59:59 +the_time_tz: in:'23:59:59-10:00' out:9:59:59 the_quoted_binary: in: 65-66-0-67-39-68- out: 65-66-0-67-39-68-<< the_escaped_binary_copy: in: 65-66-0-67-39-68- out: 65-66-0-67-39-68-<<
Reopening because it's a bug that seems to hit an increasing number of users, and we need to find a better way around it than just saying "try a different libdbi version".
*** Bug 640061 has been marked as a duplicate of this bug. ***
As already written here http://lists.gnucash.org/pipermail/gnucash-devel/2011-January/030667.html : We need to add the *detection* of this error, not (necessarily) fixing it. Currently an increasing number of users "detect" this error by observing their data loss, which is bad. Instead, we should add some code to ourselves to detect this potential error at run-time (because libdbi can change by package managers at will).
Fixed, r20147. Interestingly, I find the problem (at least on Debian Squeeze 32-bit) only happens with Sqlite3, Mike's report in comment 13 notwithstanding. The test is enabled on all the DBI backends anyway, just to be safe.
Created attachment 179143 [details] [review] Fix printf conversion specifier of 64 bit integers in message. Current SVN doesn't compile on my system, but I don't have SVN access at the moment. Attached patch fixes the printf modifier issues.
Created attachment 179633 [details] [review] Pushing the locale is done a bit earlier, before the conn_query command Using this patch standard libdbi-rpms from OpenSUSE with locale "de_DE.UTF-8" pass the Startuptest of GnuCash.
Comment on attachment 179633 [details] [review] Pushing the locale is done a bit earlier, before the conn_query command Interesting. I'll commit this when I get home this afternoon if Christian doesn't beat me to it. If the query needs to take place in the C locale then the query code in gnc-backend-dbi.c will need to be wrapped as well.
Created attachment 179641 [details] [review] Commented out locale settings around queries I tried this (without commenting it out ;-). For me it made no changes, but at the moment i don't encounter any errors at all. I will try to provoke some in the future.
It might be hard to notice errors; when I looked the other day, it seemed that most of the doubles in Gnucash are stored in KVP slots. Amounts and exchange/conversion rates are stored as pairs of int64s (numerator and denominator) to avoid rounding and binary-conversion issues.
Comment on attachment 179641 [details] [review] Commented out locale settings around queries Err... should this patch be committed or not?
Comment on attachment 179641 [details] [review] Commented out locale settings around queries Sorry, it has been already (without the extra debug statments).
Thanks. Any updates about the other commit_now attachment, attachment 179633 [details] [review] ?
Comment on attachment 179633 [details] [review] Pushing the locale is done a bit earlier, before the conn_query command Yes, also committed in r20208.
Ok, does that mean this bug is closed because we now give clear feedback, no data loss, and instructions for the user how to fix it? Thanks for the feedback!
I'm not sure I can guarantee no data loss, but I think we've covered all of the nuances of this bug.
Problem can be fixed in Ubuntu 10.10 i386 with libdbi0_0.8.3-0ubuntu1.1 - note the ".1" on the end (currently in maverick-proposed, and will be moved to maverick once more verifications of the fix come in). For details on how to test the fix from maverick-proposed see https://wiki.ubuntu.com/Testing/EnableProposed. Gnucash 2.4.2 was picking up the error in the previous version (libdbi0_0.8.3-0ubuntu1) and exiting with an error message but now with this ".1" version, Gnucash continues happily and correctly displays non-zero values.
Fix now released to Ubuntu 10.10 maverick-updates repository so should be picked up automatically by Synaptic Package Manager. Package: libdbi0_0.8.3-0ubuntu1.1 - note the ".1" on the end.
GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=611936. Please update any external references or bookmarks.