The release notes for 2.7.7 mention that gnucash now supports python-3.x.
I don't know how to test this (please tell me!) but while trying to run random files installed by gnucash with python-3.6, I saw a lot of errors:

> python3 /usr/pkg/share/gnucash/python/pycons/shell.py 

+ Trace 238494

> python3 /usr/pkg/share/gnucash/python/pycons/ishell.py 
  File "/usr/pkg/share/gnucash/python/pycons/ishell.py", line 23
    except Exception,e:
                    ^
SyntaxError: invalid syntax
> python3 /usr/pkg/share/gnucash/python/pycons/console.py 
  File "/usr/pkg/share/gnucash/python/pycons/console.py", line 82
    def seek(self, a):   raise IOError, (29, 'Illegal seek')
                                      ^
SyntaxError: invalid syntax

These errors indicate that the files only support python-2.x, not python-3.x.

Is this module still used? Then it should support python-3.x.

One other problem:

> python3 /usr/pkg/lib/python3.6/site-packages/gnucash/gnucash_business.py
* 13:51:30  WARN <gnc.engine> failed to load gncmod-backend-dbi from relative path dbi
* 13:51:30  CRIT <gnc.engine> required library gncmod-backend-dbi not found.
* 13:51:30  WARN <gnc.engine> failed to load gncmod-backend-xml from relative path xml
* 13:51:30  CRIT <gnc.engine> required library gncmod-backend-xml not found.

The backends are installed in the default location:
/usr/pkg/lib/gnucash/libgncmod-backend-xml.so
/usr/pkg/lib/gnucash/libgncmod-backend-dbi.so

Why are they not found?

The Python bindings were changed to python3 and I changed enough of the pycons demo to get it to not have any errors at GnuCash startup but didn't have time to check the rest of it. I don't know if anyone uses pycons; I actually rather doubt it, but it's not a lot of work to fix it up to work with Py3.

/usr/pkg is a default path? What distro is that? 

One doesn't run code installed in lib/pythonx directly. If any code in there even has a __main__ it's for testing that module. You'll get further with starting python, importing gnucash_business, and then calling functions added by the module. There are a bunch of example scripts in bindings/python/examples in the source directory. They're not installed anywhere. My first guess is that by trying to run gnucash_business.py directly you've not executed some setup code that enables GnuCash's binreloc code to find its modules and that regardless of whether /usr/pkg/lib is the default libdir for your distro that's not what was passed to CMake at build time (the fallback default location).

I'm packaging for pkgsrc, the default packaging system for NetBSD. /usr/pkg is the default there.

I tried running the simple test:

bindings/python/example_scripts> python3.6 simple_test.py
* 16:44:13  WARN <gnc.engine> failed to load gncmod-backend-dbi from relative path dbi
* 16:44:13  CRIT <gnc.engine> required library gncmod-backend-dbi not found.
* 16:44:13  WARN <gnc.engine> failed to load gncmod-backend-xml from relative path xml
* 16:44:13  CRIT <gnc.engine> required library gncmod-backend-xml not found.
Traceback (most recent call last):

+ Trace 238497

gnucash is setup with -DCMAKE_INSTALL_PREFIX:PATH=/usr/pkg -DCMAKE_INSTALL_LIBDIR:PATH=lib. What else do I need to set?

-DWITH_PYTHON=ON for a start. 

What does CMakeCache.txt have for LIBDIR?

I hadn't mentioned it, but I do set -DWITH_PYTHON=ON.

CMAKE_INSTALL_LIBDIR:PATH=lib
LIBDIR:STRING=lib

I realize now that simply running `python simple_test.py` from the command line doesn't access the cmake setup so it doesn't actually matter what's in CMAKE_INSTALL_LIBDIR. You need to set the paths in the environment so that dlopen can find the loadable modules.
LD_LIBRARY_PATH and GNC_LIBRARY_PATH are usually sufficient, and you can point them at either the installation path or the build path.

You don't have to do that for gnucash itself because it reads a file named etc/gnucash/environment that has the paths, but loading it isn't exposed to the bindings.

(In reply to John Ralls from comment #6)
> You don't have to do that for gnucash itself because it reads a file named
> etc/gnucash/environment that has the paths, but loading it isn't exposed to
> the bindings.

It used to be the python bindings also loaded the environment file. If this is no longer the case, that's a regression IMO. I did add code to the python binding's init block at some point for this.

I'm not at my development machine until Tuesday, so I can't check before that time.

Ah, you're right: It's exported to python via core-utils/gnc-environment.c. I thought that that was buried in gnucash-bin.c.

But the bindings don't run that on their own and there's no evidence in history that they ever did. Should they? It's a one-line addition to bindings/python/__init__.py.

I think we should.

The python bindings depend on libgnucash. As it currently stands libgnucash still contains code relying on guile. And it provides/uses libgnc-mod. Both won't work if the environment adjustments from etc/environment haven't been set.

Perhaps in the future this can be removed, but in the current context it seems required.

OK, pushed. It will be in tomorrow's release.

And reverted. Aside from it bothering Arch on Travis I found that it gets called as part of the %init in bindings/python/gnucash_core.i so calling it again from Python shouldn't be necessary.

I think the real problem is that it's not actually finding gnucash/environment because gnc_path_get_pkgsysconfdir() relies on binreloc and falls back on a compiled-in SYSCONFDIR. Binreloc isn't going to get the right answer from python so it goes to the fall-back, which is set from ${CMAKE_INSTALL_SYSCONFDIR} and that is "etc".

But there seems to be more: If I run 
   python3 /path/to/source/bindings/python/examples/simple_test.py
from the build dir it works. If I cd to the source directory and try from there it fails to find the backend modules, even if I specify GNC_MODULE_PATH and LD_LIBRARY_PATH on the command line.

(In reply to John Ralls from comment #11)
> And reverted. Aside from it bothering Arch on Travis I found that it gets
> called as part of the %init in bindings/python/gnucash_core.i so calling it
> again from Python shouldn't be necessary.
> 
Ah, I *thought* I had added it somewhere. Glad you found it.

> I think the real problem is that it's not actually finding
> gnucash/environment because gnc_path_get_pkgsysconfdir() relies on binreloc
> and falls back on a compiled-in SYSCONFDIR. Binreloc isn't going to get the
> right answer from python so it goes to the fall-back, which is set from
> ${CMAKE_INSTALL_SYSCONFDIR} and that is "etc".
> 
Let me write this out a bit more elaborately, as that helps me clear my head:

Binreloc as we use it indeed takes the base path to the running gnucash executable as the basis for all other paths. If gnucash itself is not running, this will fallback to paths defined at compile time. For linux like systems that will mostly work because on such systems the final installation path will already be defined at compile time. Not so on Windows and OS X. On these systems the installation path is only determined at installation time.

Sidenote: we currently only support python bindings on linux and similar systems. I have added the note on Windows and OS X mainly for binreloc completeness.

Obviously when using the gnucash core (which we're slowly reworking into libgnucash) via the python bindings, there is no running executable. So at that point binreloc always falls back to the fallback compile time paths. That is bound to fail.

One solution would be to adjust our binreloc code to use the base path of libgnc-core-utils instead to determine the main prefix. The comment in
https://github.com/Gnucash/gnucash/blob/unstable/libgnucash/core-utils/binreloc.c#L258
suggests binreloc used to have an alternative initialization routine to work from a library instead of an executable. I have not yet searched the net for this and don't know whether this works in a platform independent way without extra tweaks (which we also did to get it working on Windows and OS  X).

Alternatively I also found
https://github.com/gpakosz/whereami
which looks like a modernized version of binreloc. In addition to what binreloc provides is has these extras:
- it also supports ios and android which we hope to port to ourselves one day
- it can detect a library as well as an executable so we can use it to base our path initialization on libgnc-core-utils (or wherever our path initialization will eventually end up).

More generally this issue is one more reminder of how complicated path handling is when the application needs to support multiple platforms. There are many possible variations which we still may not be covering. And on the other hand it also reminds we of how scattered our path handling is in gnucash and of the many (sometimes false) assumptions it makes. Perhaps we should define a separate task to clean this up throughout the whole of gnucash. As part of this task we can decide on the best way forward. Probably not for gnucash 3.0 though.

> But there seems to be more: If I run 
>    python3 /path/to/source/bindings/python/examples/simple_test.py
> from the build dir it works. If I cd to the source directory and try from
> there it fails to find the backend modules, even if I specify
> GNC_MODULE_PATH and LD_LIBRARY_PATH on the command line.

I can't reproduce this (yet). When I try
python3 /home/janssege/Development/gnucash/unstable/bindings/python/example_scripts/simple_test.py
I get this error
Traceback (most recent call last):

+ Trace 238511

That's with today's unstable and after a clean rebuild. It looks like this error comes earlier than the errors about gncmodule loading. I'd need to figure this out first before I can comment sensibly on the difference in behaviour you describe.

MacOS actually needs to determine the load paths at runtime, which is what binreloc is all about. And there is an executable when using the python bindings: The python interpreter. The problem, of course, is that it's not necessarily in the same hierarchy as the gnucash libraries.

The compiled-in paths are relative, e.g. SYSCONFDIR is just "etc/gnucash", so they're independent of the eventual location of the hierarchy... though CMake does some funny things with paths when the install prefix is rooted in either /usr or /opt, so some care is needed.

Just to add one more layer of complication, Thomas is trying to run from DESTDIR, so the compiled in paths (e.g. ${CMAKE_INSTALL_FULL_SYSCONFDIR} is going to be wrong.

For your import problem, it's probably not finding the right "gnucash". Try adding ${builddir}/lib/python3.x/site-packages to PYTHONPATH.

No, I'm not trying to run from DESTDIR - gnucash is properly installed.

> I'm packaging for pkgsrc, the default packaging system for NetBSD. /usr/pkg is the default there.

Sorry, I took that to mean that /usr/pkg was the default DESTDIR for your packaging system. https://www.freebsd.org/cgi/man.cgi?query=hier&sektion=7&manpath=freebsd-release-ports doesn't say anything about /usr/pkg.

That's the FreeBSD man page :-)
Here's the NetBSD one: https://bugzilla.gnome.org/show_bug.cgi?id=794526

(DESTDIR is usually something like /tmp/gnucash/dest/usr/pkg/)

Really? This bug is a NetBSD manpage? ;-)

I think you meant http://netbsd.gw.com/cgi-bin/man-cgi?hier+7+NetBSD-current.

Oops. Yes, that one :-)

(In reply to John Ralls from comment #13)
> MacOS actually needs to determine the load paths at runtime, which is what
> binreloc is all about.

Indeed. And so does our Windows package. And possibly the same may be true for newer package formats on linux such as flatpak and similar.
It's safe to assume gnucash requires a runtime path discovery mechanism.

This can be binreloc though to solve scripting support we should use the alternative form to extract a prefix from a library path rather than a binary. As this is not part of our gnucash copy of binreloc I searched the web for the original sources. I found someone imported the original binreloc sources into github in 2014 but otherwise hasn't made any updates to it:
https://github.com/limbahq/binreloc
The README for this repo refers to the listaller documentation which in turn suggests they created binreloc. However the liststaller's git repo reveals the binreloc code was last touched in 2012:
https://gitorious.org/listaller/listaller/source/master:?p=listaller:listaller.git;a=history;f=contrib/binreloc;hb=HEAD

So current binreloc has two issues:
- it appears to be unmaintained
- it only targets linux and Windows (according to the liststaller documentation)

Especially the lack of OS X support is troubling. We patched our own binreloc instance to work from the executable path. I don't know if we OS X also provides similar functions to extract the full path for a shared library. Well, it probably does because whoami has figured it out.

So the alternative to binreloc would be whoami. This only handles the first part though: it extracts a full path for a binary or library (and for scripting support the latter is what we need). From there we still need to add functions to provide specific subdirectories of this path, like binreloc does.

As said before I like whoami supports our (longterm) future target platforms as well as our current ones. So for 4.0 I would definitely be interested in integrating it into libgnucash. For 3.0 that may be too radical to still accept now. But perhaps we can adjust our current binreloc sources to
- use br_init_lib rather than br_init
- extract the part of whoami that detects a library path on OS X and insert in in our binreloc code.

I haven't checked feasibility of this. I'm just looking for sensible ways forward this close to 3.0 release with minimal code impact.

For 4.0 here's one more consideration: binreloc assumes all relevant directories are subdirectories of the discovered prefix. In theory this is not guaranteed as one can arbitrarily override these directories at configure time. We have set up our Windows and OS X builds to adhere to this assumption for simplicity so on these platforms that's a non-issue. I don't know though whether there are linux like platforms that do override any directories and whether we should continue to support this. I believe currently this works if binreloc is disabled.

> And there is an executable when using the python
> bindings: The python interpreter. The problem, of course, is that it's not
> necessarily in the same hierarchy as the gnucash libraries.
> 
Indeed. I didn't mention it.

> The compiled-in paths are relative, e.g. SYSCONFDIR is just "etc/gnucash",
> so they're independent of the eventual location of the hierarchy... though
> CMake does some funny things with paths when the install prefix is rooted in
> either /usr or /opt, so some care is needed.
> 
And that explains why the script call from the build dir works. The relative paths at that point match a valid install tree (the build tree) in which the environment file can be found.

> Just to add one more layer of complication, Thomas is trying to run from
> DESTDIR, so the compiled in paths (e.g. ${CMAKE_INSTALL_FULL_SYSCONFDIR} is
> going to be wrong.
> 
> For your import problem, it's probably not finding the right "gnucash". Try
> adding ${builddir}/lib/python3.x/site-packages to PYTHONPATH.

Right, that was it. PYTHONPATH is amended in etc/gnucash/environment, but as that file isn't interpreted yet when the python script initializes my python path was wrong. Setting it properly makes the test script work.
This PYTHONPATH initialization will always be required for gnucash instances not installed in the default path and one we can't fix from within gnucash.

Our binreloc diverges rather a lot from the "official" one. Christian Stimming added Windows support to it when he set up GnuCash to use it in 2006 and I added Mac bundle support when I did the mac integration work in 2009. There's been some cleaning and fixing since as well.

It's too late to get into 3.0 but we can surely replace it during 3.x, it has no affect whatever on compatibility.

Actually, we can fix pythonpath inside of gnucash, at least for the example scripts. In each example script:

  import sys
  sys.path += GNUCASH_LIBRARY_PATH
  from gnucash import ...

Of course that requires converting the scripts to foo.py.in and running cmake's configure_file on them to insert the path.

*** Bug 795359 has been marked as a duplicate of this bug. ***

At least part of this is that we're chasing the wrong problem. The backend libraries are loaded by gnc_engine_init_part2 at https://github.com/Gnucash/gnucash/blob/maint/libgnucash/engine/gnc-engine.c#L67. Until cdcb230 that I just pushed it was passing "dbi" and "xml" as subdirectories for the respective backends; that's clearly a leftover from the old autotools builds.

With that in place I can run simple_test.py from an arbitrary directory as long as I give python3 the path.

To run from the build directory instead set GNC_BUILDDIR to point there and GNC_UNINSTALLED=1.

(In reply to John Ralls from comment #22)
> At least part of this is that we're chasing the wrong problem. The backend
> libraries are loaded by gnc_engine_init_part2 at
> https://github.com/Gnucash/gnucash/blob/maint/libgnucash/engine/gnc-engine.
> c#L67. Until cdcb230 that I just pushed it was passing "dbi" and "xml" as
> subdirectories for the respective backends; that's clearly a leftover from
> the old autotools builds.

Wow, that's a good find.

> 
> With that in place I can run simple_test.py from an arbitrary directory as
> long as I give python3 the path.

This I don't understand. Which path do you give to python3: the path to simple_test.py ? Or something else ?

python3 /path/to/simple_test.py should work if everything is installed on the normal system paths. In my MacOS jhbuild environment using a jhbuild shell is sufficient setup, binreloc understands $PREFIX; from outside of the jhbuild shell I need only add PREFIX=/path/to/installation_prefix to the environment and say $PREFIX/bin/python3...

If GnuCash isn't installed one can instead set GNC_UNINSTALLED=1 GNC_BUILDDIR=/path/to/build_directory in the environment and binreloc will find the right paths.

@Thomas Klausner please pull the latest maint and try as outlined above.

Created attachment 372753 [details]
Dockerfile reproducing issue after cdcb230 commit

Hi John,

I've attempted to try your advice with the latest maint release, but I'm still seeing the same error - it's possible I'm doing something wrong - I've attached a Dockerfile which builds from maint and runs 'import gnucash' as per your instructions but it still errors as per Thomas Klausner's original response.

If I'm doing something wrong or if you'd prefer I provide further information or a testcase in another format here please let me know.

Kind regards,

Tom

Sorry I haven't tested this yet. Is this included in the 3.2 release?

Yes.

(In reply to Tom Lofts from comment #26)
> I've attempted to try your advice with the latest maint release, but I'm
> still seeing the same error - it's possible I'm doing something wrong - I've
> attached a Dockerfile which builds from maint and runs 'import gnucash' as
> per your instructions but it still errors as per Thomas Klausner's original
> response.

No, you didn't follow my instructions.
 ln -s /build/lib/python3/dist-packages/gnucash /usr/lib/python3/dist-packages/gnucash

is not equivalent to
  ninja install

Since GnuCash *isn't* installed, you need to define GNC_UNINSTALLED and GNC_BUILDDIR:

ENV GNC_UNINSTALLED 1
ENV GNC_BUILDDIR /build

You'll get several rounds of 
WARN <gnc.core-utils> no backend loaded, or the backend doesn't define register_cb, returning 0

that you can ignore, then "OK".

For a more interesting test, substitute
RUN python3 /gnucash/bindings/python/example_scripts/simple_book.py

for your very simple test.

So I tried again with gnucash-3.2. I installed it first, then I ran:

bindings/python/example_scripts> python3 simple_test.py
* 10:50:26  WARN <gnc.engine> failed to load gncmod-backend-dbi from relative path
* 10:50:26  CRIT <gnc.engine> required library gncmod-backend-dbi not found.
* 10:50:26  WARN <gnc.engine> failed to load gncmod-backend-xml from relative path
* 10:50:26  CRIT <gnc.engine> required library gncmod-backend-xml not found.
Traceback (most recent call last):

+ Trace 238645

Looking at the system calls, I see that it is trying to open relative paths -- ${PREFIX} is missing:

 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-dbi.so"
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-dbi.dylib"
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-dbi.dylib.so"
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-dbi.dylib.la"
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-dbi.dylib"
...
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-xml.so"
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-xml.dylib"
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-xml.dylib.so"
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-xml.dylib.la"
 19079      1 python3.6 NAMI  "lib/gnucash/libgncmod-backend-xml.dylib"

I would have expected /usr/pkg/lib/gnucash/libgncmod-backend-dbi.so, which exists in the file system.

Try 
PREFIX=/usr/pkg python3 simple_test.py

What is the full path to the python3 executable on NetBSD?

No change:

PREFIX=/usr/pkg python3 simple_test.py 
* 23:31:49  WARN <gnc.engine> failed to load gncmod-backend-dbi from relative path 
* 23:31:49  CRIT <gnc.engine> required library gncmod-backend-dbi not found.
* 23:31:49  WARN <gnc.engine> failed to load gncmod-backend-xml from relative path 
* 23:31:49  CRIT <gnc.engine> required library gncmod-backend-xml not found.
Traceback (most recent call last):

+ Trace 238655

python3 is /usr/pkg/bin/python3

Are you disabling binreloc in cmake? If not, it seems broken and you're going to have to debug it.

Created attachment 372884 [details]
Updated Dockerfile with issue resolved after cdcb230 commit

(In reply to John Ralls from comment #29)

Hi John,

Thanks for your help with this - I haven't used the cmake/ninja build system much and must have missed the fact I hadn't run an install step.

I've now updated the Dockerfile to include an install step and I can confirm the issue no longer occurs for me on the 'import gnucash' line.

Thanks for getting the bug resolved and your continued development on Gnucash.

Kind regards,

Tom

GnuCash bug tracking has moved to a new Bugzilla host. The new URL for this bug is https://bugs.gnucash.org/show_bug.cgi?id=794526. Please continue processing the bug there and please update any external references or bookmarks.