I don't see any backtrace from gtkdoc itself here? Can you tell what gtkdoc command was run in the end?

I suspect it's the output of the OSError handling (sadly it doesn't say which command wasn't found), so it can't find either --cc, --ld or --run.

I'll try to build webkit, but that always takes ages here..

The command that was run was 'gtkdoc-scangobj --module=webkitdomgtk-4.0'. That doesn't work anymore.

Don't waste your time building WebKit on my behalf! Let me try to debug scangobj a bit more to see if I can figure out how exactly the command line it's executing changed. I didn't get that far last night.

What I do remember from last night is that this is the failing command:

    try:
        call(shlex.split(options.cc) + shlex.split(options.cflags) +
             ["-c", "-o", o_file, c_file])

BEFORE the patch, we had this code:

    # Compiling scanner
    command = '%s %s %s -c -o %s %s' % (options.cc, stdout, options.cflags, o_file, c_file)
    res = subprocess.check_call(command, shell=True)

Here's the value of command:

/usr/lib64/ccache/cc >/dev/null -I/home/mcatanzaro/Projects/WebKit/Source -I/home/mcatanzaro/Projects/WebKit/Source/WebKit/WebProcess/InjectedBundle/API/gtk/DOM -I/home/mcatanzaro/Projects/WebKit/WebKitBuild/GNOME/DerivedSources/webkit2gtk -I/home/mcatanzaro/Projects/WebKit/WebKitBuild/GNOME/DerivedSources/ForwardingHeaders/webkit2gtk -I/home/mcatanzaro/Projects/GNOME/install/include/webkitgtk-4.0 -I/home/mcatanzaro/Projects/GNOME/install/include/glib-2.0 -I/home/mcatanzaro/Projects/GNOME/install/lib/glib-2.0/include -I/home/mcatanzaro/Projects/GNOME/install/include/gtk-3.0 -I/home/mcatanzaro/Projects/GNOME/install/include/pango-1.0 -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/libdrm -I/home/mcatanzaro/Projects/GNOME/install/include/harfbuzz -I/home/mcatanzaro/Projects/GNOME/install/include/gdk-pixbuf-2.0 -I/home/mcatanzaro/Projects/GNOME/install/include/gio-unix-2.0/ -I/home/mcatanzaro/Projects/GNOME/install/include/atk-1.0 -I/home/mcatanzaro/Projects/GNOME/install/include/at-spi2-atk/2.0 -I/home/mcatanzaro/Projects/GNOME/install/include/at-spi-2.0 -I/usr/include/dbus-1.0 -I/usr/lib64/dbus-1.0/include -I/home/mcatanzaro/Projects/GNOME/install/include/libsoup-2.4 -pthread -I/usr/include/libxml2 -fdiagnostics-color=always -Wno-expansion-to-defined -Wno-maybe-uninitialized -Wwrite-strings -Wundef -Wpointer-arith -Wmissing-format-attribute -Wformat-security -Wcast-align -Wextra -Wall  -fno-strict-aliasing -fno-exceptions -gsplit-dwarf -Wno-unused-parameter -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_32 -Wno-cast-align -c -o webkitdomgtk-4.0-scan.o webkitdomgtk-4.0-scan.c

Now, after the patch, the command is an array:

['/usr/lib64/ccache/cc', '-I/home/mcatanzaro/Projects/WebKit/Source', '-I/home/mcatanzaro/Projects/WebKit/Source/WebKit/WebProcess/InjectedBundle/API/gtk/DOM', '-I/home/mcatanzaro/Projects/WebKit/WebKitBuild/GNOME/DerivedSources/webkit2gtk', '-I/home/mcatanzaro/Projects/WebKit/WebKitBuild/GNOME/DerivedSources/ForwardingHeaders/webkit2gtk', '-I/home/mcatanzaro/Projects/GNOME/install/include/webkitgtk-4.0', '-I/home/mcatanzaro/Projects/GNOME/install/include/glib-2.0', '-I/home/mcatanzaro/Projects/GNOME/install/lib/glib-2.0/include', '-I/home/mcatanzaro/Projects/GNOME/install/include/gtk-3.0', '-I/home/mcatanzaro/Projects/GNOME/install/include/pango-1.0', '-I/usr/include/cairo', '-I/usr/include/pixman-1', '-I/usr/include/freetype2', '-I/usr/include/libpng16', '-I/usr/include/libdrm', '-I/home/mcatanzaro/Projects/GNOME/install/include/harfbuzz', '-I/home/mcatanzaro/Projects/GNOME/install/include/gdk-pixbuf-2.0', '-I/home/mcatanzaro/Projects/GNOME/install/include/gio-unix-2.0/', '-I/home/mcatanzaro/Projects/GNOME/install/include/atk-1.0', '-I/home/mcatanzaro/Projects/GNOME/install/include/at-spi2-atk/2.0', '-I/home/mcatanzaro/Projects/GNOME/install/include/at-spi-2.0', '-I/usr/include/dbus-1.0', '-I/usr/lib64/dbus-1.0/include', '-I/home/mcatanzaro/Projects/GNOME/install/include/libsoup-2.4', '-pthread', '-I/usr/include/libxml2', '-fdiagnostics-color=always', '-Wno-expansion-to-defined', '-Wno-maybe-uninitialized', '-Wwrite-strings', '-Wundef', '-Wpointer-arith', '-Wmissing-format-attribute', '-Wformat-security', '-Wcast-align', '-Wextra', '-Wall', '-fno-strict-aliasing', '-fno-exceptions', '-gsplit-dwarf', '-Wno-unused-parameter', '-DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_32', '-Wno-cast-align', '-c', '-o', 'webkitdomgtk-4.0-scan.o', 'webkitdomgtk-4.0-scan.c']

The only difference is that the original command was a string, not an array, and the output of the original command was redirected to /dev/null. Or at least, that was attempted... I'm not sure if it works or not at that position in the command line.

(In reply to Michael Catanzaro from comment #0)
> WARNING:root:[Errno 2] No such file or directory

I'm stumped why this is happening. I would only expect it to happen if /usr/lib64/ccache/cc was not present. But it is.

No idea here either, nothing stands out. Maybe revert that patch for now until I find the time to test with webkit.

OK then, thanks for letting me revert it.

Also, the easiest way to test is to configure WebKit with 'cmake -DPORT=GTK -DENABLE_GTKDOC'. Then, after building, cd into the build directory and run 'make gtkdoc' or 'ninja gtkdoc' and watch it fail.

And the other difference was 'shell=True' in the old version, but getting rid of this is the point of the change. I'll research what other side-effect this could have. Too bad, that the os-error has no details. Michael, could you run this through: strace -e file -o strace.log <cmd> ?

(In reply to Michael Catanzaro from comment #3)
> What I do remember from last night is that this is the failing command:
> 
>     try:
>         call(shlex.split(options.cc) + shlex.split(options.cflags) +
>              ["-c", "-o", o_file, c_file])

Well that was totally wrong. I was working too late last night. :P Wasted a lot of time due to that mistake.

This is the line that's actually failing:

call(shlex.split(options.run) + ['./' + x_file])

Here's the command that gets passed:

['LD_LIBRARY_PATH=/home/mcatanzaro/Projects/WebKit/WebKitBuild/GNOME/lib:/home/mcatanzaro/Projects/GNOME/install/lib', './webkitdomgtk-4.0-scan']

So because the shell isn't used anymore, it's now trying to execute the file 'LD_LIBRARY_PATH=/home/mcatanzaro/Projects/WebKit/WebKitBuild/GNOME/lib:/home/mcatanzaro/Projects/GNOME/install/lib'. Ooops!

I don't know if you consider that a supported use of gtkdoc-scangobj --run (actually WebKit uses the RUN envvar) or not. But it looks like there's no compelling reason WebKit needs to use RUN to set LD_LIBRARY_PATH. The following WebKit patch fixes the issue:

diff --git a/Tools/gtk/gtkdoc.py b/Tools/gtk/gtkdoc.py
index 48f862a31fa..03c8e8e9b13 100644
--- a/Tools/gtk/gtkdoc.py
+++ b/Tools/gtk/gtkdoc.py
@@ -319,9 +319,9 @@ class GTKDoc(object):
             ldflags = ' "-L%s" %s ' % (self.library_path, additional_ldflags) +
 ldflags
             current_ld_library_path = env.get('LD_LIBRARY_PATH')
             if current_ld_library_path:
-                env['RUN'] = 'LD_LIBRARY_PATH="%s:%s" ' % (self.library_path, current_ld_library_path)
+                env['LD_LIBRARY_PATH'] = '%s:%s' % (self.library_path, current_ld_library_path)
             else:
-                env['RUN'] = 'LD_LIBRARY_PATH="%s" ' % self.library_path
+                env['LD_LIBRARY_PATH'] = self.library_path
 
         if ldflags:
             env['LDFLAGS'] = '%s %s' % (ldflags, env.get('LDFLAGS', ''))
@@ -332,8 +332,6 @@ class GTKDoc(object):
             self.logger.debug('CFLAGS=%s', env['CFLAGS'])
         if 'LDFLAGS' in env:
             self.logger.debug('LDFLAGS %s', env['LDFLAGS'])
-        if 'RUN' in env:
-            self.logger.debug('RUN=%s', env['RUN'])
         self._run_command(['gtkdoc-scangobj', '--module=%s' % self.module_name],
                           env=env, cwd=self.output_dir)

What do you think?

I would prefer to leave the commit from #789531 in, but also log the commands that were run in the exception. This would make it relative easy for people to fix things. If that sounds okay for you, please close this bug.

Yup, that sounds fine to me.

I guess you'll follow-up on that... note that I already reverted the commit yesterday, so you'll need to un-revert it.

Re-Reverted and pushed this:

commit f97eaa65206145c9369cdb92bfd431ab658dc8d0 (HEAD -> master, origin/master, origin/HEAD)
Author: Stefan Sauer <ensonic@users.sf.net>
Date:   Fri Nov 3 20:33:52 2017 +0100

    scangobj: log the command on error
    
    This is a followup for the change from bgo#789531. Since we don't run
    through the shell, projects can run into troubles like in bgo#789800.


Thanks everyone for debugging this.