This looks like a problem with your setup: libgstnet-1.0.la references the libgstbase-1.0.la it was linked against according to your error, but then while linking the application it doesn't find it.

You should get the same error in tests/examples/netclock

Yes, I get the same error with netclock.

However gst-inspect, gst-launch, and plugins-base/good/bad/ugly all build, and run correctly on the target system (using 1.8).

There are no build-relevant changes between 1.8.0 and GIT master for libgstnet and the netclock examples. Can you check what looks different in the .la files and the compiler/linker commandlines between both versions?

Sorry, I wasn't clear.

netclock fails to build with 1.8, but if I disable tests and examples as above then everything else works.

Is it easy to reproduce your setup? If yes, could you provide instructions for that?

To reproduce first use 64 bit Ubuntu or Debian then:

sudo apt-get install multistrap
git clone git://github.com/ali1234/gstbuild-tmp
cd gstbuild-tmp
./build

I added the compiler to my path, entered the netclock directory and ran:

strace -s 1000 -f -e execve,file make 2>&1 | grep -E 'execve|gstbase'

and this is the relevant output:

[pid 32295] execve("/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/4.9.3/../../../../arm-linux-gnueabihf/bin/ld", ["/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/4.9.3/../../../../arm-linux-gnueabihf/bin/ld", "-plugin", "/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../libexec/gcc/arm-linux-gnueabihf/4.9.3/liblto_plugin.so", "-plugin-opt=/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../libexec/gcc/arm-linux-gnueabihf/4.9.3/lto-wrapper", "-plugin-opt=-fresolution=/tmp/ccwQLBhN.res", "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_s", "-plugin-opt=-pass-through=-lpthread", "-plugin-opt=-pass-through=-lc", "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_s", "--sysroot=/home/al/gstbuild-tmp/sysroot", "--build-id", "--eh-frame-hdr", "-dynamic-linker", "/lib/ld-linux-armhf.so.3", "-X", "-m", "armelf_linux_eabi", "-o", ".libs/netclock-server", "/home/al/gstbuild-tmp/sysroot/usr/lib/arm-linux-gnueabihf/crt1.o", "/home/al/gstbuild-tmp/sysroot/usr/lib/arm-linux-gnueabihf/crti.o", "/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/4.9.3/crtbegin.o", "-L/home/al/gstbuild-tmp/sysroot/usr/lib/arm-linux-gnueabihf", "-L/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/4.9.3", "-L/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf", "-L/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc", "-L/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/4.9.3/../../../../arm-linux-gnueabihf/lib", "-L/home/al/gstbuild-tmp/sysroot/lib/arm-linux-gnueabihf", "-L/home/al/gstbuild-tmp/sysroot/lib", "-L/home/al/gstbuild-tmp/sysroot/usr/lib/arm-linux-gnueabihf", "-L/home/al/gstbuild-tmp/sysroot/usr/lib", "netclock_server-netclock-server.o", "../../../libs/gst/net/.libs/libgstnet-1.0.so", "../../../gst/.libs/libgstreamer-1.0.so", "-lgobject-2.0", "-lgmodule-2.0", "-lglib-2.0", "-lrt", "-ldl", "-lgcc", "--as-needed", "-lgcc_s", "--no-as-needed", "-lpthread", "-lc", "-lgcc", "--as-needed", "-lgcc_s", "--no-as-needed", "/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/4.9.3/crtend.o", "/home/al/gstbuild-tmp/sysroot/usr/lib/arm-linux-gnueabihf/crtn.o"], [/* 77 vars */]) = 0
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/home/al/gstbuild-tmp/gstreamer/gst/.libs/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/home/al/gstbuild-tmp/gstreamer/libs/gst/base/.libs/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/opt/vc/lib/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/lib/arm-linux-gnueabihf/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/usr/lib/arm-linux-gnueabihf/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/usr/local/lib/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/lib/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/usr/lib/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/home/al/gstbuild-tmp/gstreamer/gst/.libs/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/home/al/gstbuild-tmp/gstreamer/libs/gst/base/.libs/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/opt/vc/lib/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/lib/arm-linux-gnueabihf/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/usr/lib/arm-linux-gnueabihf/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/usr/local/lib/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/lib/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 32295] open("/home/al/gstbuild-tmp/sysroot/usr/lib/libgstbase-1.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
/home/al/gstbuild-tmp/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf/bin/../lib/gcc/arm-linux-gnueabihf/4.9.3/../../../../arm-linux-gnueabihf/bin/ld: warning: libgstbase-1.0.so.0, needed by ../../../libs/gst/net/.libs/libgstnet-1.0.so, not found (try using -rpath or -rpath-link)


So the problem here is that it is looking in the sysroot for the libs it just built, but they aren't in the sysroot because they have not been installed yet.

I also compared the .la files between a cross and a native build and I could not see any relevant differences there.

Looking closer:

/home/al/gstbuild-tmp/sysroot/home/al/gstbuild-tmp/gstreamer/gst/.libs/libgstbase-1.0.so.0

it is actually looking for ${SYSROOT}/${top_builddir} which is clearly wrong.

However if in sudo ln -s /home/al/gstbuild-tmp ${SYSROOT}/home/al/gstbuild-tmp then everything seems to build correctly.

That looks like a bug in libtool then, or you're including ${SYSROOT} in the prefix, or you didn't install with "make DESTDIR=${SYSROOT} install" but changed the prefix.

I didn't specify a prefix in this case, so it is going to default to /usr/local. Specifying a prefix makes no difference however.

It hits this error before it gets to the point where there is anything to "make install", so obviously I never ran it at all.

What do you pass to configure, i.e. how does it know about your sysroot?

see https://github.com/ali1234/gstbuild-tmp/blob/master/build

Also passing "--with-sysroot=${SYSROOT}" to configure makes no difference at all to the behaviour.

That's weird, how does it even know about your sysroot then? It should just try to install into /usr/local with this, unless it does something weird with the sysroot in LDFLAGS.

In any case, this is either a libtool bug, a toolchain bug or something wrong in your setup. There's not really anything special we're doing here

pkg-config and ld know about the sysroot because of the environment variables.

Again, "it" doesn't try to install anywhere because it crashes before that point.

Yes, it would try to install in /usr/local with this, but it simply doesn't get that far.

I now fully understand the cause of this problem. It is a well known problem with libtool, which does not properly resolve indirect linkage when using a sysroot. It isn't even related to cross compiling, as even a native build will fail if the sysroot is not "/".

The problem can be more easily demonstrated with simple gcc commands:

1. Build a shared library:
arm-linux-gnueabihf-gcc -fPIC -c libfoo.c
arm-linux-gnueabihf-gcc libfoo.o -shared -o libfoo.so

2. Build another shared library which links to the first one: 
arm-linux-gnueabihf-gcc -fPIC -c libbar.c
arm-linux-gnueabihf-gcc libbar.o libfoo.so -shared -o libbar.so

3. Build an exe which links to the second shared library:
arm-linux-gnueabihf-gcc -c exe.c
arm-linux-gnueabihf-gcc -o exe exe.o libbar.so

When libtool does these steps it sets an rpath inside the second shared library which points to the first shared library in the build dir. Then when you install the library into the prefix, libtool relinks it, adjusting the rpath.

The problem comes when ld sees the rpath on the uninstalled library while linking exe. It prepends the sysroot onto it. This would be correct behaviour for a library linked to something installed into the sysroot, but if rpath refers to the build directory then it will break. Unfortunately there is no way that ld can know what to do unless libtool tells it, which it does not.


As for a workaround: LDFLAGS="-Wl,--unresolved-symbols=ignore-in-shared-libs" will prevent ld from trying to recursively resolve the shared object links. I don't know if it has any side effects though, such as potentially making configure tests pass when they should fail.

That looks indeed like a problem in your toolchain, best to take this to the libtool developers.

For the record this bug was most recently raised on the libtool mailing list here:

http://lists.gnu.org/archive/html/libtool/2016-05/msg00022.html

(Nobody replied.)

We have the same issue here with Ubuntu 14.04 and 16.04.

This seems to be a libtool behavior change caused by a Debian/Ubuntu patch in libtool, forcing "link_all_deplibs" to "no".
See this LTTng bug report for a similar issue: https://bugs.lttng.org/issues/321

This affects all people compiling from Ubuntu with no pre-existing gstbase library in the rootfs, could this be reopened?

> We have the same issue here with Ubuntu 14.04 and 16.04.
> 
> This seems to be a libtool behavior change caused by a Debian/Ubuntu patch
> in libtool, forcing "link_all_deplibs" to "no".
> See this LTTng bug report for a similar issue:
> https://bugs.lttng.org/issues/321
> 
> This affects all people compiling from Ubuntu with no pre-existing gstbase
> library in the rootfs, could this be reopened?

This still does not sound like it's an issue with GStreamer though?

You can also use GStreamer's meson + ninja based build which does not use libtool.

(In reply to Tim-Philipp Müller from comment #20)
> > We have the same issue here with Ubuntu 14.04 and 16.04.
> > 
> > This seems to be a libtool behavior change caused by a Debian/Ubuntu patch
> > in libtool, forcing "link_all_deplibs" to "no".
> > See this LTTng bug report for a similar issue:
> > https://bugs.lttng.org/issues/321
> > 
> > This affects all people compiling from Ubuntu with no pre-existing gstbase
> > library in the rootfs, could this be reopened?
> 
> This still does not sound like it's an issue with GStreamer though?

You are right, it is not strictly speaking a GStreamer issue, but other projects have added workarounds for this libtool issue (see the LTTng link).

Another option is to build with --disable-examples --disable-benchmarks.
That's enough to avoid the issue, and does not require any hack.

> You can also use GStreamer's meson + ninja based build which does not use
> libtool.

Unfortunately, there are a lot of config options missing in GStreamer's meson.
For embedded builds, we usually disable almost all plugins, and only keep the ones we are really interested in.

Thanks for your feedback

Sorry, but I think we really don't want to start hacking around libtool issues, even less ones that are distro-specific :)

I think --disable-examples/benchmarks is a good-enough work-around for almost everyone, so in this case I think we prefer the 'do nothing' option.

The meson config option thing is on our radar, but we'd like to do it properly/nicely which will take a bit more time unfortunately. I understand if it's not an option yet for you until then of course, although I suspect these kind of things could also be hacked around temporarily either by doing patches to comment out the subdir() directives or just removing certain files again after the install step ;)

(In reply to Tim-Philipp Müller from comment #22)
> Sorry, but I think we really don't want to start hacking around libtool
> issues, even less ones that are distro-specific :)
> 
> I think --disable-examples/benchmarks is a good-enough work-around for
> almost everyone, so in this case I think we prefer the 'do nothing' option.

Yes, it is likely good-enough for many users.
I understand your position, a libtool hack would be a maintenance nightmare.

> The meson config option thing is on our radar, but we'd like to do it
> properly/nicely which will take a bit more time unfortunately. I understand
> if it's not an option yet for you until then of course, although I suspect
> these kind of things could also be hacked around temporarily either by doing
> patches to comment out the subdir() directives or just removing certain
> files again after the install step ;)

Removing files post-install would still build more stuff, but that may be balanced by the improved build speed. Thanks for the hints!

This also affects building with gobject introspection, even when examples are disabled:

g-ir-scanner: link: /bin/bash ../../../libtool --mode=link --tag=CC arm-linux-gnueabihf-gcc -std=gnu99 -o /home/al/rpi-ramdisk/packages/gstreamer/gstreamer/libs/gst/net/tmp-introspect64oX7S/GstNet-1.0 -export-dynamic -I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include -I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include/interface/vcos/pthreads -I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include/interface/vmcs_host/linux --sysroot=/home/al/rpi-ramdisk/packages/sysroot/ -pipe -march=armv7-a -marm -mthumb-interwork -mfpu=neon-vfpv4 -mtune=cortex-a7 -mabi=aapcs-linux -mfloat-abi=hard --sysroot=/home/al/rpi-ramdisk/packages/sysroot/ tmp-introspect64oX7S/home/al/rpi-ramdisk/packages/gstreamer/gstreamer/libs/gst/net/tmp-introspect64oX7S/GstNet-1.0.o -L. libgstnet-1.0.la -L../../../gst -L/home/al/rpi-ramdisk/packages/sysroot//home/al/rpi-ramdisk/packages/gstreamer/gstreamer/gst/.libs -L/home/al/rpi-ramdisk/packages/sysroot//usr/lib/arm-linux-gnueabihf -Wl,--export-dynamic -lgmodule-2.0 -pthread -lgstreamer-1.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0
libtool: link: arm-linux-gnueabihf-gcc -std=gnu99 -o /home/al/rpi-ramdisk/packages/gstreamer/gstreamer/libs/gst/net/tmp-introspect64oX7S/.libs/GstNet-1.0 -I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include -I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include/interface/vcos/pthreads -I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include/interface/vmcs_host/linux --sysroot=/home/al/rpi-ramdisk/packages/sysroot/ -pipe -march=armv7-a -marm -mthumb-interwork -mfpu=neon-vfpv4 -mtune=cortex-a7 -mabi=aapcs-linux -mfloat-abi=hard --sysroot=/home/al/rpi-ramdisk/packages/sysroot/ tmp-introspect64oX7S/home/al/rpi-ramdisk/packages/gstreamer/gstreamer/libs/gst/net/tmp-introspect64oX7S/GstNet-1.0.o -Wl,--export-dynamic -pthread -Wl,--export-dynamic  -L. ./.libs/libgstnet-1.0.so -L../../../gst -L/home/al/rpi-ramdisk/packages/sysroot//home/al/rpi-ramdisk/packages/gstreamer/gstreamer/gst/.libs -L/home/al/rpi-ramdisk/packages/sysroot//usr/lib/arm-linux-gnueabihf -lgmodule-2.0 /home/al/rpi-ramdisk/packages/gstreamer/gstreamer/gst/.libs/libgstreamer-1.0.so -lgio-2.0 -lgobject-2.0 -lglib-2.0 -pthread -Wl,-rpath -Wl,/home/al/rpi-ramdisk/packages/gstreamer/gst/lib
/home/al/rpi-ramdisk/packages/toolchain/bin/../lib/gcc/arm-linux-gnueabihf/4.9.3/../../../../arm-linux-gnueabihf/bin/ld: warning: libgstbase-1.0.so.0, needed by ./.libs/libgstnet-1.0.so, not found (try using -rpath or -rpath-link)
./.libs/libgstnet-1.0.so: undefined reference to `gst_byte_writer_init_with_data'
collect2: error: ld returned 1 exit status
linking of temporary binary failed: Command '['/bin/bash', '../../../libtool', '--mode=link', '--tag=CC', 'arm-linux-gnueabihf-gcc', '-std=gnu99', '-o', '/home/al/rpi-ramdisk/packages/gstreamer/gstreamer/libs/gst/net/tmp-introspect64oX7S/GstNet-1.0', '-export-dynamic', '-I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include', '-I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include/interface/vcos/pthreads', '-I/home/al/rpi-ramdisk/packages/sysroot//opt/vc/include/interface/vmcs_host/linux', '--sysroot=/home/al/rpi-ramdisk/packages/sysroot/', '-pipe', '-march=armv7-a', '-marm', '-mthumb-interwork', '-mfpu=neon-vfpv4', '-mtune=cortex-a7', '-mabi=aapcs-linux', '-mfloat-abi=hard', '--sysroot=/home/al/rpi-ramdisk/packages/sysroot/', 'tmp-introspect64oX7S/home/al/rpi-ramdisk/packages/gstreamer/gstreamer/libs/gst/net/tmp-introspect64oX7S/GstNet-1.0.o', '-L.', 'libgstnet-1.0.la', '-L../../../gst', '-L/home/al/rpi-ramdisk/packages/sysroot//home/al/rpi-ramdisk/packages/gstreamer/gstreamer/gst/.libs', '-L/home/al/rpi-ramdisk/packages/sysroot//usr/lib/arm-linux-gnueabihf', '-Wl,--export-dynamic', '-lgmodule-2.0', '-pthread', '-lgstreamer-1.0', '-lgio-2.0', '-lgobject-2.0', '-lglib-2.0']' returned non-zero exit status 1
Makefile:1041: recipe for target 'GstNet-1.0.gir' failed

I finally figured out the cause of this. It is due to a Debian patch in libtools, and it was reported 14 years ago - I just updated that bug with my findings:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=297726