GNOME Bugzilla – Bug 570342
Temporary .xml2po.mo breaks parallel builds
Last modified: 2009-10-26 15:15:11 UTC
Please describe the problem: When building documentation like gnome-user-docs in parallel, i.e. passing -j2 or similar to make, then there is a high chance of the build creating wrong documents or failing altogether. This is because gnome-doc-utils.make calls xml2po with "-p *.po" as an argument, causing xml2po to write the formatted messages to a file ".xml2po.mo" in the current working directory. As this file is the same for multiple parallel tasks, it is likely that one will write to it before or while another is reading it. As a solution, the conversion from *.po to *.mo using msgfmt could be moved from xml2po to gnome-doc-utils.make, and xml2po could be called with "-t *.mo" instead of "-p *.po". In this way, there would be no shared temporary files, and parallel builds would succeed. Steps to reproduce: Repeatedly build gnome-user-docs with "make -j10" Actual results: With some probability, the build will fail. Error messages will likely include "Can't open MO file '.xml2po.mo'" as well as some form of UnicodeDecodeError due to wrong or truncated input. Expected results: Documentation built in parallel, without errors, and resulting output identical to a completely serial build. Does this happen every time? No, but a loop of 10 repeats usually was enough to reproduce it here. Other information: Gentoo bug report with patch: https://bugs.gentoo.org/255479
Created attachment 127831 [details] [review] Avoid .xml2po.mo Patch against current gnome-doc-utils trunk to have make file create the *.mo files so that xml2po won't have to use .xml2po.mo.
Adding gnome@gentoo.org for tracking purpose.
Danilo, any thoughts on this?
In #593175, I've proposed a patch so as xml2po creates a real temporary file (tempfile.mkstemp()) instead of the pseudo temporary .xml2po.mo I suppose it would resolves this issue, wouldn't it?
(In reply to comment #4) > I suppose it would resolves this issue, wouldn't it? I assume it would, yes. The solution offered here wouldn't conflict with patch from bug #593175, though, so there would be no harm in implementing both, or either, depending on which patch gets approved of sooner. In theory, the cached *.mo files might be useful for debugging some rare build issues, but on the other hand the temporary files will probably conserve disk space during the build. Neither argument is really strong.
Martin, now that the patch in #593175 has been committed, could you check if this issue seems to be resolved?
I think we should still do this regardless of #593175. It should speed things up when there are multiple top-level XML files for each language, since we'll only create an mo file once, instead of once for each file. And with Mallard documents, we're only going to see more and more top-level XML files.
Comment on attachment 127831 [details] [review] Avoid .xml2po.mo Fine, I've committed the patch, slightly adapted because there has been some changes to gnome-doc-utils.make in between.
Created attachment 145577 [details] [review] Clean up generated *.mo files in clean-doc-lc Thanks for committing this! Once I realized the project had moved from svn to git, I even found the commit. :-) I guess we should also take care of the cleanup code mentioned in bug #593175 comment #4. Probably easiest if one simply adds _DOC_MOFILES to the list, as the attached patch does.
Right, something is still needed to properly clean the mo files. However, your patch doesn't look good because this list of files are tested in a loop and only deleted if not existent in the source dir (which is sensible for po files, but not for mo files). What about simply adding a "rm -f $(_DOC_MOFILES)" just before this loop?
(In reply to comment #10) > What about simply adding a "rm -f $(_DOC_MOFILES)" just before this loop? Agreed. I guess I got a bit confused by the fact that the old cleanup for the temporary .xml2po.mo files checks for existence before removing. Somehow I thought we'd have to do the same for the new mo files. Should there be some kind of comment indicating that the .xml2po.mo cleanup code is legacy and could be removed one day?
I've applied the fix and a comment as you suggested. http://git.gnome.org/cgit/gnome-doc-utils/commit/?id=984063e7a4189b31cff99e61736d01eb5d3906f6