GNOME Bugzilla – Bug 328652
Autotranslation of gedit-test-utf8-procedural-api.py fails with gedit-2.13
Last modified: 2006-02-01 17:36:23 UTC
Running CVS pyspi and dogtail as of 24th January 2006 with gedit-2.13.1-2 on Fedora (rawhide, development to FC5). [dmalcolm@cassandra examples]$ LANG=ja_JP.UTF-8 ./gedit-test-utf8-procedural-api .py Creating logfile at /tmp/dogtail/logs/gedit-test-utf8-procedural-api_20060125-17 3747 ... Detecting distribution: Red Hat/Fedora/derived distribution Bonobo accessibility support initialized GTK Accessibility Module initialized /tmp/dogtail/screenshot_20060125-173801.png Screenshot taken: /tmp/dogtail/screenshot_20060125-173801.png click on {child with name="Save" ("保存(S)", "保存")} Translation not found for "Save As…" searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 3) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 4) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 5) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 6) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 7) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 8) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 9) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 10) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 11) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 12) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 13) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 14) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 15) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 16) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 17) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 18) searching for child of Node roleName='application' name='gedit' description='': "Save As…" dialog (attempt 19) Translation not found for "Save as..." searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 3) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 4) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 5) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 6) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 7) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 8) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 9) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 10) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 11) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 12) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 13) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 14) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 15) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 16) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 17) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 18) searching for child of Node roleName='application' name='gedit' description='': "Save as..." dialog (attempt 19) Traceback (most recent call last):
+ Trace 65611
focus.dialog('Save as...')
else: raise FocusError, name
Problem seems to be in: gedit/gedit-commands-file.c: file_save_as specifies the trasnslatable string in the source file as _("Save As\342\200\246"). "Save As\342\200\246" is an octal encoding of a NUL-terminated UTF-8 encoded string consisting of "Save As" followed by the U+2026 HORIZONTAL ELLIPSIS unicode character. Looking in the packaged mo file I see the same encoded string. hexdump -C /usr/share/locale/ja/LC_MESSAGES/gedit.mo | grep -C 3 "Save As" 00008e10 20 31 31 00 53 61 6e 73 20 38 00 53 61 6e 73 20 | 11.Sans 8.Sans | 00008e20 52 65 67 75 6c 61 72 20 31 31 00 53 61 6e 73 20 |Regular 11.Sans | 00008e30 52 65 67 75 6c 61 72 20 38 00 53 61 72 64 69 6e |Regular 8.Sardin| 00008e40 69 61 6e 00 53 61 76 65 00 53 61 76 65 20 41 73 |ian.Save.Save As| 00008e50 e2 80 a6 00 53 61 76 65 20 5f 41 73 2e 2e 2e 00 |....Save _As....| 00008e60 53 61 76 65 20 61 6c 6c 20 6f 70 65 6e 20 66 69 |Save all open fi| 00008e70 6c 65 73 00 53 61 76 65 20 74 68 65 20 63 68 61 |les.Save the cha| Looks like it's failing to locate the translation in the call to gettext.dgettext(). What is the encoding to strings passed to GNU gettext, or what rules cover this? The test currently searches for that unicode string directly.
oddly enough, es_ES.UTF-8 and fr_FR.UTF-8 still work for me.
I saw that issue before while working on Bug 320561. Managed to get around it using ugettext instead of dgettext. IMHO, I think ugettext would handle unicode better. Found another example regarding python with gettext as well. <http://mail.python.org/pipermail/python-dev/2003-April/034511.html> Hope it helps.
At first I couldn't find this 'uggettext' thing you were talking about. I eventually found it. The way we're currently doing things is: <figure out what translation domains we need> gettext.dgettext(domain, stringToTranslate) The way we'd need to use ugettext, unless I'm wrong: <figure out what mo-files we need> trans=gettext.GNUTranslations(open(fileName)) trans.ugettext(stringToTranslate) Which will require a slight refactoring of dogtail.i18n. Or am I missing something?
Yes, I think it would involve some changes for dogtail to handle unicode. Please try if it helps. <http://bugzilla.gnome.org/attachment.cgi?id=54256&action=view> + # ensure that there is actually mo exist file that locale + if (os.path.isfile(fullPath)): + gettext.bindtextdomain (self.domainName, "/usr/share/locale/") + gettext.textdomain (self.domainName) + t = gettext.translation(self.domainName, "/usr/share/locale/") + _ = t.ugettext + result = _(srcName)
OK, it looks like I've got a fix hacked up using ugettext(), with a net loss of 20 lines - yay! I happened to fix two other bugs, also - yay! I forgot to attach the patch before I committed, so here's the link to the commit: http://cvs.gnome.org/bonsai/cvsquery.cgi?branch=&dir=dogtail&who=zcerza&date=explicit&mindate=2006-01-26%2020:54&maxdate=2006-01-26%2020:56
Unfortunatly there looks to be a regression here :-( I just tested CVS HEAD on my Fedora rawhide box. A judicious "print moFile" statement in GettextTranslationDb __init__ shows that it's loading the moFile for every single language, not just the one under test (and it does it for every package in the dependenct chain, so it's a lot of mo files...) LANG=ja_JP.UTF-8 ./gedit-test-utf8-procedural-api.py Creating logfile at /tmp/dogtail/logs/gedit-test-utf8-procedural-api_20060126-212739 ... Detecting distribution: Red Hat/Fedora/derived distribution /usr/share/locale/af/LC_MESSAGES/gedit.mo /usr/share/locale/am/LC_MESSAGES/gedit.mo [ lots of mo files snipped ] /usr/share/locale/mn/LC_MESSAGES/GConf2.mo /usr/share/locale/ms/LC_MESSAGES/GConf2.mo /usr/share/locale/nb/LC_MESSAGES/GConf2.mo /usr/share/locale/ne/LC_MESSAGES/GConf2.mo Traceback (most recent call last):
+ Trace 65652
dogtail.i18n.loadTranslationsFromPackageMoFiles('gedit')
load(packageName, getDependencies)
translationDbs.append(GettextTranslationDb(moFile))
self.__gnutranslations = gettext.GNUTranslations(open(moFile))
self._parse(fp)
plural = v[1].split('plural=')[1]
This fatal error is isolatable to a simple reproducer: Python 2.4.2 (#1, Dec 17 2005, 13:02:22) [GCC 4.1.0 20051214 (Red Hat 4.1.0-0.9)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import gettext >>> gettext.GNUTranslations(open("/usr/share/locale/ne/LC_MESSAGES/GConf2.mo")) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/gettext.py", line 177, in __init__ self._parse(fp) File "/usr/lib/python2.4/gettext.py", line 301, in _parse plural = v[1].split('plural=')[1] IndexError: list index out of range Do you want me to file separate bugs for: (i) the multilanguage loading thing? (ii) the mo file that can't be loaded? It may be that various translation files are broken (e.g. the specific one above), and that's causing the problem, but if so the loading of excessive numbers of language files is more likely to trigger that bug. Grrrr :-(
Grrr indeed. Well, I just committed a fix that filters the list of mo-files so that we only get the language we want. Unfortunately there's still more to do, it seems...
Created attachment 58491 [details] [review] final nail in the coffin This patch should be the final nail in the coffin of this bug. Lots of encoding-related fixed.
Er, "fixes". Anyway, I just committed the patch. I'm closing the bug because all my testing went fine, but feel free to reopen it if you find more problems.
I did find some problem with the patch. # LANG=ja_JP.UTF-8 ./gedit-test-utf8-procedural-api.py Creating logfile at /tmp/dogtail/logs/gedit-test-utf8-procedural-api_20060201-024349 ... Detecting distribution: Red Hat/Fedora/derived distribution Segmentation fault
(In reply to comment #10) Well, no python code should ever cause the python interpreter to crash. This has to be either python's, pyspi's, or some other C-written module's fault. Maybe the rpm bindings?