GNOME Bugzilla – Bug 127049
building reference documentation fails when builddir != srcdir
Last modified: 2011-03-29 10:18:35 UTC
I'm using the latest cvs versions of gtk-doc and glib2. I'm using automake-1.7.6 and autoconf-2.57. I've searched bugzilla, but I don't think thes has come up with glib yet. Here's the situation: If I run configure and make from the topsrcdir, documentation is generated as expected, so I *know* that gtk-doc and its deps are correctly installed. The problem is if you try to build glib2 from a dir other then the source tree, which I often do to prevent polluting the source tree with generated files (which makes diff'ing easy). What happens is the sgml/xml source files are assuming that config.status generated files will appear in the same tree, which isn't true if builddir != srcdir. So obviously the process complains about not being able to find some files and quits. Also isn't able to find definitions for a bunch of cpp macros from what I guess are generated headers(which wasn't the case when builddir == srcdir). Then it finally bails while trying to run gtkdoc-mkhtml, saying that it can't find ../version.xml (which I note is a config.status generated file). I haven't devised a means of fixing, however I do have some suggestions. I'm not sure if the gtk-doc utilities are capable of doing it, but the documentation really ought to be built in the builddir, not the srcdir. This is the behaviour of automake when you build pdf's or html from texinfos. If needs be, any static sgml/xml source dirs should be passed to jade as a full canonical path. I think any use of ellipse paths (i.e. ../../) ought to be discouraged.
Mostly a gtk-doc problem, since gtk-doc.make comes from there. Although the version.xml problem has to be solved here.
Note that the tmpl file regeneration must occur in the sourcedir since they are checked into CVS, so a "clean" system that never does anything to the sourcedir isn't possible.
the template file updating is another of gtk-doc's irritating perks. We regularly get conflicts for no apparent reason in GStreamer. I'm starting to feel it would be better to have something like an update rule that would update the tmpl files. Or, alternatively, change the gtk-doc makefile snippets so that the gtk-doc documentation build gets done in a build subdirectory, where the original templates get copied, and then updated. Someone updating the gtk-doc templates in CVS would then manually diff back to the original tmpl/ files. This would also avoid CVS conflicts. It would also be easy to then add a rule to check for diffs when doing a docs build, so it would tell you what needs updating. If this sounds like a good thing to have, I wouldn't mind spending some work on a template gtk-doc project tarball that worked this way. What do you think ?
Sounds useful to me.
A `make distcheck` has a build directory that differs from the source directory. So all `make distcheck` are effected by this bug.
crossreference to bug 156643 (proposes an update on the make file)
Note that glib, gtk+ 'make distcheck' successfully, and have DISTCHECK_CONFIGURE_FLAGS = --enable-gtk-doc, so I'm actually a little suprised by this bug report.
Owen, the reason that glib and gtk+ pass make distcheck is because they have a horrible hack that makes the srcdir writable :) this hack is in gtk-doc.make scan-build.stamp: $(HFILE_GLOB) $(CFILE_GLOB) @echo '*** Scanning header files ***' @-chmod -R u+w $(srcdir) ... @echo '*** Rebuilding template files ***' @-chmod -R u+w $(srcdir) and so on. It's a little messy. I spent some time on the GStreamer doc build to correctly work for out of source builds and in distcheck, I should probably work on getting that into gtk-doc.make.
(In reply to comment #7) > Note that glib, gtk+ 'make distcheck' successfully, and have > DISTCHECK_CONFIGURE_FLAGS = --enable-gtk-doc, so I'm actually > a little suprised by this bug report. The reason is that these packages distribute version.xml (they add it to variable content_files). It's an ugly hack to distribute a configure-generated file. See also bug 314105, which proposes to pass the version of the module via a --stringparam option.
Here's my own addition to the list of hacks to accommodate this: content_files = \ $(top_srcdir)/docs/reference/your-module/version.xml $(top_srcdir)/docs/reference/your-module/version.xml: cp $(top_builddir)/docs/reference/your-module/version.xml $@ This is nonintrusive; i.e., it doesn't require modification of gtk-doc.make. And I suspect it will work with a read-only srcdir from a tarball build (though I haven't tried it). I think it's entirely reasonable to require a writable srcdir for bootstrapping.
(In reply to comment #10) > I think it's entirely reasonable to require a writable > srcdir for bootstrapping. I can agree with this part. It's relatively straightforward to rewrite gtk-doc.make to properly build in builddir instead of messing with srcdir. However, there are several issues. 1. Templates. Sigh. Files that cannot decide whether they want to be human-created or generated always break everything. Personally, I do not care -- if you employ a fundamentally flawed approach, VPATH builds won't work for you. 2. Content files. The best approach is to keep them where they are (srcdir) and instruct the output format builder to look there. See bug 460753. Makefile.am has to list content files as $(srcdir)/foo.xml. 3. Location of foo-docs.sgml. Passing something like $(abs_srcdir)/foo-docs.sgml to xsltproc screws up relative locations. What I currently do is to list $(srcdir)/$(DOC_MAIN_SGML_FILE) in rule dependences, but before HTML is built, copy the driver to the build directory test -f Makefile.am || cp -f $(srcdir)/$(DOC_MAIN_SGML_FILE) . (Makefile.am is assumed to be always present in srcdir and never in builddir. This is a bit silly, but I could not find a reliable portable test whether builddir and srcdir resolve to the same directory.) 4. Probably locations of foo.sections, foo.types. I use --rebuild-types --rebuild-sections and therefore normally get them in builddir as expected. However, people editing them manually may have to use a rule similar to what is suggested in comment 10. 5. Location of foo-overrides.txt. The same as above. See also 6. 6. False bootstrapping. When gtk-doc tools do not see some files, they think they are run for the first time and `helpfully' create them. Namely foo-docs.sgml and foo-overrides.txt. Maybe the tools should gain a new option: the directory to look for human-written files in (which files, that depends on --rebuild-* options given). The files will be written only if they are not present in this source directory. 7. Source code in builddir. Sometimes generated source files are documented (think version.h). This means one has to process $(DOC_SOURCE_DIR) both under $(srcdir) and $(builddir), i.e. both have to be given as --source-dir. However, one must not give a directory twice as --source-dir, because it would be scanned twice, leading to lots of WARNING: Structure has multiple definitions: FooBar and subsequently a total breakage. Currently I do if test -f Makefile.am; then \ x=; \ else \ x=--source-dir=$(top_builddir)/$(DOC_SOURCE_DIR); \ fi; \ gtkdoc-foo --source-dir=$(top_srcdir)/$(DOC_SOURCE_DIR) $x ... but this is inelegant. Perhaps the tools should be just made smart enough to avoid processing a directory twice. 8. Installation and making dists. We have to decide whether to copy files from srcdir or builddir. Many things that will appear in builddir are also distributed and can be installed or packaged without rebuilding them. I currently test whehter we have the files available in builddir and fall back to srcdir if they are not found: d=; \ if test -s html/index.sgml; then \ echo 'gtk-doc: Installing HTML from builddir'; \ d=html; \ elif test -s $(srcdir)/html/index.sgml; then \ echo 'gtk-doc: Installing HTML from srcdir'; \ d=$(srcdir)/html; \ else \ echo 'gtk-doc: Nothing to install'; \ fi; \ ...
I was thinking about 7. Deciding whether two read-only directories are backed by one storage or not is generally impossible (think nonlocal mounts, bind mounts, ...). So I can see two approaches: 1. Ignore the really obscure cases and use Cwd::realpath() for directory comparison. If they are different, assume the directories are different. 2. Reformulate the problem as `what we actually want to avoid is scanning the same file twice', use Digest::MD5 and skip files we have already seen. Which makes more sense?
Created attachment 92597 [details] [review] proposed patch Point 7, ignore source files that have been already scanned in gtkdoc-scan and gtkdoc-mkdb. This enables to safely pass both srcdir and builddir as --source-dir to these tools.
Created attachment 92611 [details] [review] proposed patch Points 3, 4, 5 and 6, add --master-dir option to gtkdoc-scan, gtkdoc-scangobj, gtkdoc-mktmpl, gtkdoc-mkdb. I have not tested it very thoroughly, but I have a more-or-less kluge-free gtk-doc.mk that passes distcheck (really, not by cheating) using --master-dir, so it at least sort of works. Note this leaves gtk-doc in a bit schizophrenic state, as we have options both for the output directory and the (possibly read-only) input directory now. The former were written with the notion of building in srcdir and writing files elsewhere, but AFAIK this have never worked properly. The latter assume we are building in builddir which makes much more sense to me.
(Forgot to note patch 92611 does not implement --master-dir in gtkdoc-scanobj, I suppose no one uses this anyway nowadays). The remaining issues: 1. Cheerfully ignored. 2. This is addressed by bug 460753 + simple change people have to do in their Makefile.am. 7. The way the stock gtk-doc.make uses DOC_SOURCE_DIR from Makefile.am (--source-dir=$(DOC_SOURCE_DIR)) permits to pass multiple directories only with a hack: DOC_SOURCE_DIR=$(top_srcdir)/some/dir --source-dir=$(top_builddir)/some/dir I'm not sure how to fix this without breaking compatibility too much. 8. and the other gtk-doc.make stuff. This is missing. I can offer some working gtk-doc.make examples building in builddir, but I'm not sure I am able to write a works-for-everyone-is-compatible-with-everything gtk-doc.make. Fortunately, only the tool changes are crucial. Software authors typically do not ship gtk-doc, but they can (and often do) tailor gtk-doc.make, so the tool changes at least make possible reasonable builds, gtk-doc.make can be changed later.
The --master-dir patch has issues with old-style builds. That is, a new makefile would specify --master-dir, but if users don't update their old makefiles it should work at least as before, not worse. Must default $MASTER_DIR to ".".
Created attachment 93480 [details] [review] proposed patch New --master-dir patch, defaults $MASTER_DIR to "." and fixes a few typos.
Yeti, totally forgot about this. I guess the problem still persists and this is the only way to do it (yet another option). Do you still have this as a local change? If so could you regenerate the patch?
Nope, can't find it locally, the VCS has changed too many times since... The new option should be more or less internal to gtk-doc (i.e. it can be set from what the makefiles already provide). The patch makes VPATH build possible on the tools' side for the template-less style, i.e. the tools learn to distinguish between source and build directories. It must be then accompained with some makefile changes to make it actually work (not present in the patch, only some suggestions in comments). Without the makefile changes it should behave as bad as before but that is not a problem (if it doesn't break even more). However, I don't really remember how the new option works...
Created attachment 158121 [details] [review] a new attempt for out-of-srcdir builds Instead of all the previous black magic I propose this patch. Basically it copies all shipped files to builddir (if builddir!=srcdir) and builds there. This removes the hack with the "@-chmod -R u+w $(srcdir)". It seems to work fine for me, do I overlook something?
Stefan, I tried your patch #158121 (sligtly modified to be merged with my custom makefile) and "make distcheck" gives lot of these errors: cp: impossibile creare il file normale `/home/nicola/sandbox/adg/adg-0.5.6/_build/docs/cpml/cpml-docs.xml': Permission denied cp: impossibile creare il file normale `/home/nicola/sandbox/adg/adg-0.5.6/_build/docs/cpml/cpml-docs.xml': Permission denied cp: impossibile creare il file normale `/home/nicola/sandbox/adg/adg-0.5.6/_build/docs/cpml/cpml-overrides.txt': Permission denied cp: impossibile creare il file normale `/home/nicola/sandbox/adg/adg-0.5.6/_build/docs/cpml/cpml.types': Permission denied Furthermore, distcheck failed with: ERROR: files left in build directory after distclean: ... a list of all files under ./docs/cpml/html/ ... ./docs/cpml/html.stamp ./docs/cpml/cpml-docs.xml ./docs/cpml/sgml.stamp ./docs/cpml/cpml.types ./docs/cpml/cpml-overrides.txt This is the relevant directory content after "make distcheck": ~/sandbox/adg/adg-0.5.6/_build/docs/cpml$ ll drwxr-xr-x 2 nicola users 752 8 apr 19:24 html -r--r--r-- 1 nicola users 964 8 apr 19:24 cpml-docs.xml -r--r--r-- 1 nicola users 0 8 apr 19:24 cpml-overrides.txt -r--r--r-- 1 nicola users 0 8 apr 19:24 cpml.types -rw-r--r-- 1 nicola users 10 8 apr 19:24 html.stamp -rw-r--r-- 1 nicola users 9 8 apr 19:24 sgml.stamp
Created attachment 170754 [details] [review] a new attempt for out-of-srcdir builds This seems to be more complete. I still need to figure a better way to support gtkdoc-check. Right now one needs to also edit the own Makefile.am and remove the "cd $(srcdir) &&" part. Unfortunately I need something backwards compatible like going from: TESTS_ENVIRONMENT = \ DOC_MODULE=$(DOC_MODULE) DOC_MAIN_SGML_FILE=$(DOC_MAIN_SGML_FILE) \ cd $(srcdir) to something like this (but that does not work): TESTS_ENVIRONMENT = \ DOC_MODULE=$(DOC_MODULE) DOC_MAIN_SGML_FILE=$(DOC_MAIN_SGML_FILE) \ if test -r $(srcdir)/$(DOC_MODULE)-undocumented.txt ; then \ testdir=$(srcdir); \ else \ testdir=$(builddir); \ fi; cd $$testdir &&
Have a fix for gtkdoc-check too, will push when git.gnome.org is back.
Created attachment 170927 [details] [review] a new attempt for out-of-srcdir builds The gtkdoc-check issue is fixed in git. This new patch works for tmpl and non-tmpl based projects. I can use it in own projects (buzztard), have applied it to gstreamer core, but there are still some issues (e.g. in glib with HTML_IMAGES).
Created attachment 170929 [details] [review] a new attempt for out-of-srcdir builds trying with the right file. Also the glib problem is a bug in the glibs Makefile.am.
Created attachment 170930 [details] [review] a new attempt for out-of-srcdir builds try really hard to attach the right patch and take the next bus :/
Created attachment 176577 [details] [review] fix out of srcdir builds A similar patch has been pushed a few days agao for the modified gtk-doc build in gstreamer and works fine. I'll test this a little more, but will push this next year. Please test.
Comment on attachment 176577 [details] [review] fix out of srcdir builds Aparently I committed this somehow :/ and its now part of the 1.16 release. Thanks everybody for *not* testing.