GNOME Bugzilla – Bug 439654
Aqbanking in gnucash 2.1.1 under Windows cannot download balance/transactions via HBCI
Last modified: 2018-06-29 21:37:14 UTC
=== Short description === When using aqbanking-tool to download balance/transactions from a HBCI account, the download fails. The same happens when using the integrated aqbanking in gnucash. Error messages from aqbanking-tool (full messages in the detailed description below): 3:2007/05/19 13-12-37:gwen(948):netlayer.c: 1607: Not connected 3:2007/05/19 13-12-37:gwen(948):netlayer.c: 1476: Errors on all netlayers 3:2007/05/19 13-12-37:aqhbci(948):dialog.c: 340: Could not connect to bank (-1) 3:2007/05/19 13-12-37:aqhbci(948):dialog.c: 394: Error sending message for dialog Unable to send (network error) I do not get the balance or list of transactions. === Detailed description/how to reproduce === I installed gnucash-2.1.1-setup.exe (2007-04-28 16:40) from sourceforge. I use Windows XP Professional with all automatic updates. First, I configured an HBCI test account with aqhbci-tool following the instructions on http://wiki.gnucash.org/wiki/AqBanking: --- START INTERACTION --- C:\Program Files\gnucash\bin>aqhbci-tool addmedium -t pintan 5:2007/05/19 13-01-01:aqbanking(5172):banking.c: 1895: Configuration file "C:\Documents and Settings\unruh\.banking\settings.conf" does not exist, will create it later. 3:2007/05/19 13-01-02:gwen(5172):db.c: 1802: Error opening file "C:\Documents and Settings\unruh\.banking\backends\aqhbci\settings.conf": No such file or directory Medium added. C:\Program Files\gnucash\bin>aqhbci-tool adduser -m 0 -s www.hora-obscura.de/pintan/PinTanServlet -b 80007777 -u gnucash C:\Program Files\gnucash\bin>aqhbci-tool getsysid ===== Certificate Received ===== The following certificate has been received: Name : www.hora-obscura.de Organisation: www.hora-obscura.de Department : GT90185227 Country : DE City : unknown State : unknown Valid after : 2007/02/15 14:28:06 Valid until : 2008/05/17 14:28:06 Hash : 69:CF:A0:D9:EA:1F:4B:DA:5F:FA:7D:EE:75:87:C9:FF Status : Certificate is valid Do you wish to accept this certificate? (1) Yes (2) No Please enter your choice: 1 1 ===== Certificate ===== Do you want to accept this certificate permanently? (1) Permanently (2) This session only (3) Abort Please enter your choice: 1 1 5:2007/05/19 13-09-21:aqbanking(5524):banking.c: 5681: User accepted certificat permanently 3:2007/05/19 13-09-21:aqhbci(5524):job.c: 193: Job "JobTan" not supported by your bank ===== Enter Password ===== Please enter the password for PINTAN-20070519-130102 Input: 12345 ***** 3:2007/05/19 13-09-51:aqhbci(5524):dialog.c: 628: Could not disconnect from bank (-6) --- END INTERACTION --- So far, everything is as I expected. Then I tried to get the balance of the first account using aqbanking-tool: --- START INTERACTION --- C:\Program Files\gnucash\bin>aqbanking-tool request --balance C:\Program Files\gnucash\bin>aqbanking-tool exec AqHBCI started Connecting to bank... Resolving hostname "www.hora-obscura.de" ... IP address is 82.165.27.189 5:2007/05/19 13-12-32:aqbanking-tool(948):banking.c: 5553: Automatically accepti ng certificate "69:CF:A0:D9:EA:1F:4B:DA:5F:FA:7D:EE:75:87:C9:FF" Connected. Selecting iTAN mode "iTAN-Verfahren (PV 1)" Opening dialog Encoding queue ===== Enter Password ===== Please enter the password for PINTAN-20070519-130102 Input: 12345 ***** Sending queue Waiting for response Response received HBCI: 0020 - Dialoginitialisierung erfolgreich (M) HBCI: 3920 - Erlaubte Signatur-Verfahren (S) Bank message received Bank message received Encoding queue Sending queue 3:2007/05/19 13-12-37:gwen(948):netlayer.c: 1607: Not connected 3:2007/05/19 13-12-37:gwen(948):netlayer.c: 1476: Errors on all netlayers 3:2007/05/19 13-12-37:aqhbci(948):dialog.c: 340: Could not connect to bank (-1) 3:2007/05/19 13-12-37:aqhbci(948):dialog.c: 394: Error sending message for dialog Unable to send (network error) Disconnecting from bank... Disconnected. 3:2007/05/19 13-12-37:aqhbci(948):outbox.c: 1368: Error performing queue (-6) AqHBCI finished. Executing Jobs: 3 of 3 3:2007/05/19 13-12-37:aqbanking-tool(948):banking.c: 965: No progress context open --- END INTERACTION --- I expected the last command to output something like the following (as is done when issuing the same command on a Gentoo machine): accountInfoList { ... Somewhere here the balance occurs ... } # accountInfoList Similarly, when trying to get the list of transactions, I get: --- START INTERACTION --- C:\Program Files\gnucash\bin>aqbanking-tool request --transactions C:\Program Files\gnucash\bin>aqbanking-tool exec AqHBCI started Connecting to bank... Resolving hostname "www.hora-obscura.de" ... IP address is 82.165.27.189 5:2007/05/19 13-16-24:aqbanking-tool(5872):banking.c: 5553: Automatically accept ing certificate "69:CF:A0:D9:EA:1F:4B:DA:5F:FA:7D:EE:75:87:C9:FF" Connected. Selecting iTAN mode "iTAN-Verfahren (PV 1)" Opening dialog Encoding queue ===== Enter Password ===== Please enter the password for PINTAN-20070519-130102 Input: 12345 ***** Sending queue Waiting for response Response received HBCI: 0020 - Dialoginitialisierung erfolgreich (M) HBCI: 3920 - Erlaubte Signatur-Verfahren (S) Bank message received Bank message received Encoding queue Sending queue 3:2007/05/19 13-16-28:gwen(5872):netlayer.c: 1607: Not connected 3:2007/05/19 13-16-28:gwen(5872):netlayer.c: 1476: Errors on all netlayers 3:2007/05/19 13-16-28:aqhbci(5872):dialog.c: 340: Could not connect to bank (-1) 3:2007/05/19 13-16-28:aqhbci(5872):dialog.c: 394: Error sending message for dialog Unable to send (network error) Disconnecting from bank... Disconnected. 3:2007/05/19 13-16-28:aqhbci(5872):outbox.c: 1368: Error performing queue (-6) AqHBCI finished. Executing Jobs: 3 of 3 3:2007/05/19 13-16-28:aqbanking-tool(5872):banking.c: 965: No progress context open --- END INTERACTION --- I expected: accountInfoList { ... long list of transactions ... } # accountInfoList === Further analysis === I reproduced the problem on another Windows XP Professional machine. I found a bug report on a newsgroup that seems to cover the same problem on Mac OS X (http://arcknowledge.com/finance.aqbanking.devel/2006-08/msg00037.html in German). That bug report proposes to add a sleep(1) instruction in the GWEN_Socket_Write function in src/os/posix/inetsocket.c in the gwenhywfar source. I did this as follows: GWEN_ERRORCODE GWEN_Socket_Write(GWEN_SOCKET *sp, const char *buffer, int *bsize){ int i; assert(sp); assert(buffer); assert(bsize); #ifdef OS_DARWIN /* this is just a temporary ugly hack for OS X, this has to be investigated * further */ if (sp->haveWaited==0) { sleep(1); sp->haveWaited=1; } #endif sleep(1); (Last line is my added sleep instruction.) I tried this with gwenhywfar 2.5.4 as well as with 2.9.0beta. I compiled gwenhywfar and aqbanking under cygwin and the above problem did not occur any more (balance/transactions downloaded successfully). Alternatively, I tried to enable the "if (sp->haveWaited==0) { sleep(1); sp->haveWaited=1; }" snippet that was already present in the source (see above) instead of inserting an unconditional sleep(1), this did not work. I failed to build aqbanking under MinGW or to compile the whole of gnucash, so I cannot verify whether the hotfix works with the aqbanking client integrated into gnucash.
Thanks for the report and detailed analysis. However, the distributed aqbanking version is compiled with mingw and not cygwin, hence it uses the code in src/os/windows and not src/os/posix. Your report has been forwarded to Martin Preuss and he will keep an eye on this when he's programming aqbanking3. Nevertheless I have used the transaction download from the test server successfully at least once, so I'm not so sure whether this is a systematical error. And again, if you still try to use mingw, I hope gnucash's install.sh should help you in compiling the necessary prerequisites.
*** Bug 463448 has been marked as a duplicate of this bug. ***
Created attachment 96770 [details] [review] Add the unconditional sleep(1) to the windows code @Dominique: I'm still hesitating to add an unconditional sleep(1) here, because on some Windows systems this obviously works fine already. Can you confirm whether this adds a noticable delay in the network connections? Or is the connection just working fine? Attached is the patch that I'd include in gwenhywfar and release a gwen-2.6.2, if that patch looks good.
I think I have the same problem, cannot download transactions anymore om win xp with GnuCash 2.2.1 and aqhbci 2.3.2. I could post detailed log messages if needed. But I like to try the patch for the gwenhywfar. I think I might manage to compile GnuCash using the guide at http://wiki.gnucash.org/wiki/Windows. But I have two questions how to apply this patch. I do not know if this is the right place to ask them. If not please excuse me! Do I need to compile the whole GnuCash package or is their a way to just recompile the libgwenhywfar-38.dll. How to I apply the patch? Do I have to copy the code attached to this bug into the source code or do I link this patch in otherwise during comilation. You see I have never applied a patch before although I did some programming and compilation.
> @Dominique: I'm still hesitating to add an unconditional sleep(1) here, because > on some Windows systems this obviously works fine already. Can you confirm > whether this adds a noticable delay in the network connections? Or is the > connection just working fine? Since posting the bug, I switched to a new computer. Since I have not installed a suitable build environment since, I cannot test your patch. However, let me add some further information on the bug. On my new computer (Thinkpad T43p, Win XP SP2) the bug still occurs. When I am at home (network: DSL over wireless), the transaction download rarely (or never?) succeeds. When I am at work (network: University network over ethernet) the transaction download often succeeds. So probably the occurrence of the bug depends on the timing of the network. This might explain why many people use aqbanking without problems under Windows. Helge: What network do you use (wireless/ethernet, DSL/telephone/etc)? Yours, Dominique.
@Dominique: I use actually two different internet connections, company LAN and mobile broadband with 3G (UMTS). With both I have the same problems. Log messages look also the same for me. /Helge
@Helge: Indeed you do not need to compile *all* of gnucash; recompiling of libgwenhywfar-38.dll is sufficient. You can run the install.sh script of gnucash until it downloaded and installed the mingw compiler components. Then, unpack the gwenhywfar tarball, apply the patch, have gwenhywfar compiled (with appropriate configure options as listed in install.sh) and copy the resulting libgwenhywfar-38.dll (in src/.libs) into the gnucash/bin directory to overwrite the previous libgwenhywfar-38.dll. That would be enough for testing this.
@Christian: I tried that but did not make it. I ended up with just object files in the .libs directory (like gwenhywfar.o). After running the install.sh script as well as downloading and patching the gwenhywfar-2.6.1 source I run these commands: $ ./configure --host=mingw32 --with-openssl-includes=/c/soft/openssl/include --disable-binreloc ssl_libraries=-L/c/soft/openssl/lib ssl_lib=-leay32 -lssl32 --prefix=/c/soft/gwenhywfar CPPFLAGS=-I /c/soft/regex/include -I /c/soft/gnome/include LDFLAGS=-L/c/soft/regex/lib -L/c/soft/gnome/lib -lintl make make check make install The make command got me a message: ".../inetsocket.c:1275: undefined refernce to 'sleep'" Then I tried to run the full install.sh script. I added a pause so I could apply the patch to the source code. I get the same error: c:/soft/tmp/gwenhywfar-2.6.1/src/os/windows/inetsocket.c:1293: undefined reference to `sleep' Did I forget some reference?
@Helge: Oh, this "undefined reference" error indeed says I didn't prepare the source code change correctly. Turns out the function name must be "_sleep" instead of "sleep" on mingw windows (i.e. with leading underscore). Hence, please try again but the new function call, instead of sleep(1); must be _sleep(1); This should result in the file src/.libs/libgwenhywfar-38.dll that was mentioned above.
@Christian: Compilation worked fine now, thanks for your help! I got some errors again but the libgwenhywfar-38.dll was created. I just tried aqhbci-tool from the command line but did not succeed. I will try again later and enable logging. From GnuCash though it worked now again to get transactions. But it also seems to work with the original dll.
Hm... with gnucash it worked with the old and the new libgwenhywfar-38.dll? In comment#4 you said it didn't work with the old libgwen. If anyone can tell me "It didn't work with the old (original) libgwenhywfar-38.dll but now it works with the self-compiled new one", then I'd immediately make sure to release a new libgwen version...
If anyone sends me a compiled version of libgwenhywfar-38.dll I will compare the behaviour with that shipped with the most current stable release of gnucash. I will compare both in terms of download success as well as connection speed. (You get my email by hovering over my user name in the comment (only when logged in.) If possible, send a link to the lib instead of the whole lib (because of the file size).
I finally managed to compile gwenhywfar on my windows machine. For me, the _sleep(1) didn't solve the problem. I tried _sleep(100) and now it works. I successfully downloaded transactions and submitted a transaction using PIN/iTAN. I have not done any performance measures - for me it's okay like this. I may try to reduce the sleep to e.g. 5 the next days... for now I'm really happy to use gnucash on windows! @Dominique: I sent you 2 links for a dll with _sleep(1) and and another one with _sleep(100) (strip the "_sleep_x" from the filename after downloading)
@Christian: Sorry for my unclear and confusing comment#10. From the command line I only tried 'aqhbci-tool getsysid …', got some error messages and stopped. Later I saw that these online accounts were though created successfully, maybe even alreaday some days ago. I could map them to a account in GnuCash's online banking setup. Getting online transactions from GnuCash has worked before but only sometimes. Yesterday it worked again with one bank from my office thru the company LAN. It did not work however later from home thru mobile broadband, both with original and patched dll. Today I compiled the libgwenhywfar-38.dll again but now with sleep(100) and made some tests. With that version I can download transactions from all my banks and also from the test account (80007777). With the original dll and the one patched with sleep(1) however it works only with one out of three banks and the test account. Sometimes I do not even reach the point where I am asked for my PIN. This again is thru company LAN. I will try via mobile broadband in the evening. Thanks for the tip using sleep(100), Timm!
Now I tried from home via mobile broadband. No success at all downloading any transactions. :-( What can I do? Create log files and post them? Or try to read them myself? Or compile another libgwenhywfar-38.dll withe an even longer sleep time?
Just compiled the libgwenhywfar-38.dll with _sleep(500). Now I get transactions from all banks even thru mobile broadband. :-) Is it ok to use such a high value or is there a risk now for running into other trouble. 500 means 500 ms, doesn't it?
Oh uh - *milli*seconds? On Linux, the sleep() function without the underscore uses the argument as *seconds*. For that reason I was rather surprised to read about _sleep(100) here... @Helge: As these are 500ms, we could indeed use _sleep(500). Does that introduce any noticable delay? If this fixes your problem *and* there isn't any noticable or annoying delay, I'm ready to include this in a new release.
This is now in the gwenhywfar SVN. @Helge: Can you check again whether the 500ms delay isn't a problem anywhere? If there is no problem, I will make a new release this weekend.
@Christian: The delay is noticeable, it seems like the communication stops two or three times while downloading transactions. But it could also be that it is really waiting for the reply of the bank server. But it took all in all less than 30 s to download like 40 transactions. And I surely prefer waiting some seconds to entering all transactions from my bank account by hand. :-) I also tried to send two transactions. Worked also fine. I will work more with GnuCash over the weekend. But this sleep delay does not seem to do anything bad. Is the libgwenhywfar-38.dll also used also elsewhere beside down- and uploading transctions? If you tell me how to delete a medium and a user in the aqhbci-tool I could to the whole test process starting from "aqhbci-tool addmedium -t pintan" again. Now that I got it running I of course also like to understand what the story behind is. Can anyone give me a hint on that? Where in the process have I added this sleep? And what is the real problem, is the connection terminated by the aqhbci-tool if the bank server is not responding fast enough? I read http://osdir.com/ml/finance.aqbanking.devel/2006-08/msg00037.html but that was also referring to another post which I cannot find.
I did a few tests with the DLLs compiled by Timm. Both files (1ms & 100ms) do not lead to success in my home network. However, when I raise the delay to 500ms (hexeditor), I can successfully download the transactions. (It failed once in about five tries.) The additional delay is noticeable, but not terrible. I did a transaction download which took between 5 and 10 secs (hard to guess the exact time since there is a password input involved on the way) for about 30 transactions from Deutsche Bank. I am not sure how long it takes without the delay. I also did a test with delay 10000ms. In this case the download took about 130 sec. Thus I think that the delay is called about 12-13 times, so the total delay in the 500ms case should be about 6 secs. @Christian: Indeed, the sleep(1) call in my original post was meant to sleep for one second. However _sleep and sleep have different semantics (and the latter seems only to be available on cygwin). @Helge: I suppose the "other post" is http://ml.osdir.com/finance.aqbanking.devel/2006-02/msg00032.html and follow-ups (but I am not sure, I found it by googling sleep(1) aqbanking). However, all these posts are concerned with MacOS, not Windows. My best guess why the problem occurs is the following: After some socket operation (open/send/etc) the socket is not immediately ready for some OS specific reasons. When using the socket immediately, the call will return some value that is interpreted by libgwen as a failure. So to really fix the problem, one would first have to find which is the exact call that fails (this should be traceable, since it finally results in the log message "network error"). Then one could either understand what is returned and why (probably just some slight misinterpretation of the Windows-API by the libgwen or MingW programmers). Or, failing that, one could try and add the sleep only when the socket operation fails. I.e., replace "socketop()" by "if (!socketop()) { _sleep(1000); socketop() }" This would still be a hotfix, but - assuming it works - it would at least restrict the additional delay to users that need it. However, keep in mind that the above are all educated guesses.
@Dominique: Thanks for the explanation, I couldn't have explained this better - I also can only guess. The "exact call that fails" is the send() in inetsocket.c:1281; however, I haven't ever programmed sockets myself, let alone on Windows, let alone using mingw. I can only guess on Windows some other call has to be checked before using send()... whatever. If the _sleep() is necessary, we'll add it here. Just a last question: Do you think we can reduce the 500ms to 300ms and would it still work? I'd feel better with 300ms...
Oh, and if anyone is up for yet further testing, you can make the _sleep() into a conditional one that is used only on first call of the GWEN_Inetsocket_Write; see src/os/posix/inetsocket.c:527; this would need the additional member variable in inetsocket_p.h.
> Just a last question: Do you think we can reduce the 500ms to 300ms and would > it still work? I'd feel better with 300ms... I just tried it five times and it worked every time with 300ms. However, the whole thing is highly nondeterministic. It may just be that my network has a good day and that it will need a larger delay tomorrow. Or that other computers with other network connections will need a larger delay. So 300ms is a bit borderline (since 100ms certainly is too little) but for me it is OK. (If it should turn out that it still is too little, one might still increase the delay in the next revision). @Timm+Helge: How does 300ms work on your systems? @Christian: Thanks a lot for your work.
@Dominique: Since on my system it already works with 100ms, 300ms would be okay for me. @Christian: I just tried using a conditional _sleep(500) like you suggested in comment #22, and it did NOT work on my system. Another thing: stdlib.h states, that the "_sleep"-function is deprecated and should be replaced by the Win-API-function "Sleep". I am compiling a version with "Sleep(500)" to test whether this will work, too.
@Christian: Yes, it works using "Sleep" as well. So if you want you may use this instead of "_sleep" in case mingw sometime decides to strip the _sleep-function.
@Dominique: Just tried with _sleep(300) and Sleep(300), both worked fine. 200 ms though does not work.
I seem to have the same problem, could you send me the dll to jmechtel on gmail.com ? I use two different HBCI accounts and they both stopped working after I moved and have a new internet connection (same provider, different router). I might be able to manage to patch myself, but the dll itself should work no? Thanks so much for help I am glad this problem can be solved Jan
@Jan: I have just sent you the libgwenhywfar-38.dll patched with Sleep(300). Hope it will help you.
I seem to have the same problem, could you send me the dll, too. (ganter at ojojo.de) I am new to gnucash and not experienced with compiling software. I hope the new .dll will work for me. Grüße aus Mannheim Axel
@Axel: I will send the libgwenhywfar-38.dll patched with Sleep(300) right away. Hope it works also in Mannheim!
Created attachment 97633 [details] libgwenhywfar-38.dll patched with Sleep(300) This seems to help in most cases to fix this bug. Extract this dll to the GnuCash-installation\bin\ directory and replace the original dll.
Yes it works, thanks a lot !!
What is the status here, especially as of GnuCash 2.2.5?
Works for me with Gnucash 2.2.5 - if I remember correctly Christian added the call to "Sleep" some time ago
Good answer ;-) Marking as FIXED. Thanks to anyone involved!
GnuCash bug tracking has moved to a new Bugzilla host. This bug has been copied to https://bugs.gnucash.org/show_bug.cgi?id=439654. Please update any external references or bookmarks.