GNOME Bugzilla – Bug 603432
EBCDIC tests fail without iconv or ICU support
Last modified: 2017-11-04 14:52:22 UTC
OS X 10.4/powerpc, trying to build libxml2-2.7.6 (note, "Version" field of bugzilla only goes up to 2.7.3). libiconv is supplied by fink. I patched libxml2 to use a custom catalog dir instead of /etc (to avoid collision of my personal build with the system official one). My existing xmllint and xsltproc are having problems, so I avoid detecting them. XMLLINT=/usr/bin/true XSLTPROC=/usr/bin/true ./configure --prefix=/sw --mandir=/sw/share/man --with-iconv=/sw --without-python --disable-dependency-tracking --with-html-dir=/sw/share/doc --with-html-subdir=libxml2 --with-history [no problems] make [some warnings but build completes] make check ./runtest && ./testrecurse && ./testapi && ./testchar&& ./testdict && ./runxmlconf File ./test/ebcdic_566012.xml generated an error File ./test/ebcdic_566012.xml generated an error File ./test/ebcdic_566012.xml generated an error Result for ./test/ebcdic_566012.xml failed File ./test/ebcdic_566012.xml generated an error Result for ./test/ebcdic_566012.xml failed File ./test/ebcdic_566012.xml generated an error Result for ./test/ebcdic_566012.xml failed File ./test/ebcdic_566012.xml generated an error Failed to parse ./test/ebcdic_566012.xml File ./test/ebcdic_566012.xml generated an error Failed to parse ./test/ebcdic_566012.xml File ./test/ebcdic_566012.xml generated an error Failed to parse ./test/ebcdic_566012.xml File ./test/ebcdic_566012.xml generated an error Failed to parse ./test/ebcdic_566012.xml File ./test/ebcdic_566012.xml generated an error ## XML regression tests [lots more ## tests, but no other failures] Total 2819 tests, 10 errors, 0 leaks make: *** [runtests] Error 1
All tests passed using the same build process for 2.7.3 then broke as of 2.7.4
Of course it passes on Linux x86_64 (and i686 as far as I can tell). Can you valgrind runtest to see if it shows something ? or try to valgrind xmllint ./test/ebcdic_566012.xml This is OS specific, as far as I can tell it's not available freely for my platform and I have no way to test it, so I rely on you to pursue the issue. The test was added in 2.7.4 IIRC so it's not a regression. Daniel
% ./xmllint test/ebcdic_566012.xml test/ebcdic_566012.xml:1: parser error : encoding not supported EBCDIC Well that's a pretty good reason for the test of ebcdic parsing to fail. Where in the code is a place this encoding could get disabled? Or is it delegated to some external component?
It's delegated to iconv, weird ... though there is a lot of different flavours of EBCDIC encoding, which is one of the problem the patches and tests tried to check. Daniel
Same results when building with either OS X 10.4's libiconv (1.9) or with fink's (1.12) and also with xmllint from OS X 10.4 itself. I dove into encoding.c a bit, and see that xmlGetCharEncodingHandler is calling xmlFindCharEncodingHandler with each of three variants of the ebcdic encoding name ("EBCDIC", "ebcdic", "EBCDIC-US"). Side note, given that xmlFindCharEncodingHandler does case-insensitive testing, does xmlGetCharEncodingHandler really need to try separately with lc and uc forms? In xmlFindCharEncodingHandler, no suitable handlers are found, and then iconv_open is failing (icv_in==-1 and icv_out==-1 even after trying name and upper. iconv and charset encoding is apparently "nonstandard" on darwin. My system /usr/lib/charset.alias has no entries and fink doesn't even install one because someone said darwin doesn't use it properly anyway.
libiconv-1.13.1 is latest release, and the only mention of ebcdic is in its NOTES file: Q: Support EBCDIC ? A: No! d'oh:(
Seems like libiconv EBCDIC support is only implemented in glibc's onboard iconv http://sourceware.org/ml/libc-ports/2007-09/msg00006.html whereas the standalone GNU libiconv does not support it, does not intend to support it, and does not have an open plugin mechanism by which third-party could supply an implementation. Maybe libxml2 might need a --disable-ebcdic configure flag? Or if there an autoconf test that can distinguish GNU libiconv from glibc iconv, that could at least disable the self-test that is apparently a hopeless case on that type of platform?
This appears to be a 32-bit vs. 64-bit issue, and apparently only affects test/ebcdic_566012.xml. I can still reproduce these errors with "make check" if I force libxml2.dylib (and xmlllint) to run in 32-bit on Intel hardware (i386 architecture).
(In reply to David Kilzer from comment #8) > This appears to be a 32-bit vs. 64-bit issue, and apparently only affects > test/ebcdic_566012.xml. > > I can still reproduce these errors with "make check" if I force > libxml2.dylib (and xmlllint) to run in 32-bit on Intel hardware (i386 > architecture). I am wrong. This is the difference between using iconv vs. ICU for encoding support with libxml2. The version of libxml2 that ships with OS X is compiled to use ICU, not iconv, and ICU supports the EBCDIC encodings (while iconv does not). I made the mistake of compiling my 32-bit binaries without specifying --with-icu, thereby causing this bug to appear while I was testing.
I think this can be closed as "RESOLVED INVALID" (or whatever the moral equivalent of "Behaves Correctly" is). If you want EBCDIC support, configure libxml2 --with-icu instead of --with-iconv on Mac OS X.
Ideally, the tests should be skipped if there's no EBCDIC support. This has bugged me for quite some time when developing on OS X.
*** Bug 680542 has been marked as a duplicate of this bug. ***
Fixed here: https://git.gnome.org/browse/libxml2/commit/?id=4b4135977e82b7c9d3bba87a24fb7b5609312e14