GNOME Bugzilla – Bug 697133
2.4.12 regression: livelock while importing OFX
Last modified: 2018-06-29 23:14:33 UTC
NB: This bug is against version 2.4.12, but Bugzilla didn't give that as an option at the time of filing. After updating to 2.4.12, importing an OFX file generated by Fidelity causes GnuCash to go into an infinite loop. I reverted to 2.4.11 and the file imported fine. I tried using a fresh data file in 2.4.12 (with just the default account templates and no transactions), but the import still hung. Attaching a debugger to the process showed 4 threads, three of which were blocked on system calls but one was seemingly always stuck in libstdc___6!ZNSi3getEPcic+0x1a. Here's the backtrace:
+ Trace 231714
Here's a detailed trace of the last couple of frames, lightly sanitised (lowercase xxx's are mine, preserving the length of strings). Without knowing anything about this code, three things stand out: all of the filename_* variables are junk / apparently uninitialized; file_is_xml = false (it is XML, but maybe it's also uninitialised); and buffer starts with a \0.
+ Trace 231715
@Geert: Could that be because of your libofx patch, committed by Benoit on 2012-11-26, "Patch by Geert Janssens to allow lines longer than 1024 caracters" Our 2.4.12 binary comes with libofx-0.9.6 that contains that patch, but our 2.4.11 still had libofx-0.9.2 without that patch.
I'm experiencing hanging with OFX import after updating to 2.4.12 on Windows XP Pro SP3 32bit. File > Import > Import OFX/QFX - Select file > Import. The Generic import transaction handler hangs and I have to kill the process. Reverted back to 2.4.11 and OFX import works again.
Thanks for reminding about the missing version number! Number created and version of this bug adjusted.
(In reply to comment #0) > After updating to 2.4.12, importing an OFX file generated by Fidelity causes > GnuCash to go into an infinite loop. Most likely this is caused by the change in the used ofx import library "libofx". I've switched from version 0.9.2 to 0.9.6 in the gnucash-2.4.12 Windows binary release. We would therefore have to find out how to fix this in the libofx library. Can you attach a (small) example file that causes this problem? Thanks! Can you or someone else here verify whether the same problem occurs on Linux as well (if libofx-0.9.6 is used there)? If it's a general libofx problem, the same problem should already be visible with libofx's internal command line tool "ofxdump".
For the first time I actually built GnuCash in Ubuntu 12.04. It identified itself as release 2.4.12. Then I tried an OFX import and it succeeded. I have not tried an OFX import yet in Widows.
Windows - sorry, I think.
(In reply to comment #5) > Can you or someone else here verify whether the same problem occurs on Linux as > well (if libofx-0.9.6 is used there)? If it's a general libofx problem, the > same problem should already be visible with libofx's internal command line tool > "ofxdump". I have libofx 0.9.4 on Fedora 15 and have no problem with imports.
I see a couple of folders named libofx 0.9.4 on my laptop.
I was about to try to create an anonymised OFX file to demonstrate the problem, when I noticed that my failing OFX file was all on one very long (4000-character) line, and also lacked a trailing linebreak. Simply inserting a few line breaks in the file was enough to fix the problem and get the import process started. It sounds like whatever patch was applied to libofx to handle long lines (a) is broken and (b) wasn't always necessary before (since this file imports fine in 2.4.11). Do you still need me to construct a test-case, or is this information enough to debug the problem? I suspect that if you took any sufficiently-large OFX file and removed _all_ the line-breaks you'd hit the same problem.
Ok, I've been able to reproduce this here by using an OFX file with no linebreaks at all. I've released libofx-0.9.7 that fixes that bug. All you still need is a gnucash windows binary that contains libofx-0.9.7 instead of 0.9.6. It might take up to 7 days until we've created the next gnucash windows binary. Please watch the files at http://code.gnucash.org/builds/win32/2.4/ for a setup.exe with revision of at least 22896 or larger - the last file from April 8th (r22872) still has the old erroneous libofx.
Ditto on hanging: System: Windows 7 Pro on HP Probook 4525 Gnucash: 2.4.12 svn r22850 on 2013-03-31 Bug Process: File>Import>OFX/QFX Generic Import Transaction Import Window pops up blank, and hangs. Tried importing as both OFX and QIF on 3 bank and credit card account files. Hangs every time, and need to stop process, and restart Gnucash. Will install 2.4.11
Everyone with this problem (on windows): Please download http://code.gnucash.org/builds/win32/2.4/gnucash-2.4.12-svn-r22905-setup.exe (which is a 2.4.12 version plus this bugfix), test it and tell us whether this problem is fixed. If this bug is fixed and everything else is fine, we'll release a 2.4.13 of that state.
oops, I just noticed this build probably still contains the erroneous libofx-0.9.5 (as can be verified in $prefix/lib/pkgconfig/libofx.pc). Hence, no bugfix available in this version. It will come later, sorry.
*** Bug 698066 has been marked as a duplicate of this bug. ***
ok, here you are: Please download http://code.gnucash.org/builds/win32/2.4/gnucash-2.4.12-svn-r22907-setup.exe (which is a 2.4.12 version plus this bugfix, now really), test it and tell us whether this problem is fixed. Thanks!
Created attachment 241666 [details] tracefile ofx import failure
Created attachment 241667 [details] failed import ofx file This is a slightly obfuscated copy of the file that choked Gnucash. I used a binary editor to change the user name and account number. I did not use Notepad because it wanted to change the character of the line endings. The original file has LF only(0A).
I just looked at all my recent traces going back to the beginning of this month and found each and every one to start with the same qof error, so it probably does not relate to the import failure. Sorry.
@David Carlson: You didn't say which operating system you're on - above you said Ubuntu, which isn't the topic of this bug. Please add more comments here only if you also run into the bug with gnucash-2.4.12 and libofx-0.9.6 which makes gnucash hang on importing some ofx files. If you encountered a different behaviour or have a different combination of OS and libofx, please open a separate bug. Thanks!
Sorry, I was not clear. I am able to use either OS, but in this case I am referring to Windows 7 64 bit. I keep my primary copy of my data file accessible to the computer that is running Windows 7, so I needed to take that computer back to 2.4.11 when it failed to accomplish the OFX imports. Right after I reverted to 2.4.11 again, the next trace only had the first line but none of the following lines that you see in the attachment with comment 17. I did not check to see which libofx came in with r22907.
I did check the splash screen to see that the r22907 build was the version that was actually running in W 7 when the import choked. Hopefully the windows installer deletes all the dependencies listed in the windows uninstaller file and replaces them with the correct dependencies. I bring that up because the trace files that I am looking at came from a setup made a long time ago when release 2.4.6 came out. I was surprised that numerous installs later it is still working. Sometimes issues have arisen with other software when new versions do not completely overwrite old ones.
r22907 works for me. Thanks!
Sorry, not for me. build r22907 Windows 7 pro: 32 QFX still hangs. QIF: doesn't hang but receives following error message: "Line 3: File does not appear to be in QIF format: C* Read aborted."
(In reply to comment #24) @Colleen: Any chance to attach a sample QFX file here?
Created attachment 241770 [details] Another QFX file that chokes R22907 Hello again. In Windows 7 64 bit environment: I first deleted GnuCash 2.4.11 Then I re-installed gnucash-2.4.12-svn-r22907-setup.exe Then I looked in Windows 7 OS\program files(x86)\gnucash\bin and observed that there was a program named libofx-5.dll dated 4/15/3013 Then I launched Gnucash and saw r22907 in the splash screen. I opened a copy of a backup file dated April 4, 2013. I verified that there were no transactions in the target account that would be in the the test import. I attempted to import a qfx file dated April 5, 2013. The generic import window opened and stalled, still empty. After waiting a reasonable time, I aborted the program. This is a slightly different qfx from the previous one, but it is from the same FI. I have attached a sanitized copy with the original line breaks (0A) intact.
Maybe the dll had a 2013 date.
@David: Thanks for your first test file, I can reproduce the "hang" also on Linux with libofx-0.9.7. Seems like the empty line is the problem - if you remove that one, the file works fine. I'll see how to fix this in the libofx code.
Created attachment 241774 [details] credit card ofx that chokes r22907 Here is an ofx from a credit card account that also chokes rr22907
I've released an updated libofx-0.9.8 that fixes the problems with the files uploaded here. The bugfix will be available in any gnucash r22911 or newer. Please watch http://code.gnucash.org/builds/win32/2.4/ (2.4 branch) or http://code.gnucash.org/builds/win32/trunk/ (trunk branch, any build from April 17th or newer) for the next available windows binary. Once any of those is available (which might take up to 2-3 days), please report here whether the files work again.
The credit card file has (0D)(0A)(0D)(0A) before the <OFX> tag and removing (0D)(0A) and leaving (0D)(0A)is not sufficient to allow the import to work. Thus the file cannot be fixed using Wordpad. That file has (0D)(0A) throughout. As you stated, removing one of the two (0A)'s before the <OFX> tag in the ALEC file works for my case too.
Christian, thank you for working on ofx, particularly for fixing the code I added :( I haven't had time to chime in until now. I looked at your changes, which seem all very decent. There is one additional thing I wondered about. On the mailing list Dave Carlson suggested that on Windows we may experience some CRLF vs LF issues as well (1). I realize that the ofx code only checks for "\n" (line feed). Shouldn't we also be stripping "\r" (carriage return) ? It appears ofx files can have a mixture of CRLF and LF line endings. (1) https://lists.gnucash.org/pipermail/gnucash-user/2013-April/048741.html
The "ofxdump" test utility of libofx-0.9.8 is able to read all three supplied OFX files (both ALEC and Target) and extract the transactions correctly. Hence, I'd like to know whether http://code.gnucash.org/builds/win32/2.4/gnucash-2.4.12-svn-r22912-setup.exe fixes this issue on windows.
After I found out that Firefox changed the way that it shows that downloads are happening I actually did download and install r22912. It succeeded in importing those files on this Windows 7 64 bit system. Your fix works for me. Thanks.
Ok, we'll release a gnucash-2.4.13 with the fixed libofx library. Thanks everyone!
Reassign version to 2.4.x so that individual 2.4 versions can be retired.
GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=697133. Please update any external references or bookmarks.