GNOME Bugzilla – Bug 78007
i18n input methods, font selection, display all broken
Last modified: 2004-12-22 21:47:04 UTC
The gnome-terminal neither displays Korean characters properly, nor accepts them by X input method(XIM). With the previous versions of gnome 1.4, I can input Korean characters but they're garbled regardless of font settings. It has been fixed since the recent red-carpet update, but the problem revived with gnome 2.0. I'm using ami as my input method application. Could you please help me with this problem? Thanks.
When I open another terminal from gnome-terminal, it shows following error message. The font "-sony-fixed-medium-r-normal-*-16-*-*-*-c-*-iso8859-1" does not support all the required character sets for the current locale "ko_KR.eucKR"(Missing character set "KSC5601.1987-0") If I select a Korean font in my terminal profile, terminal window expands with garbled texts and displays below message. The font "-baekmuk-batangbdf-medium-r-normal-*-16-*-*-*-m-*- ksc5601.1987-0" does not support all the required character sets for the current locale "ko_KR.eucKR"(Missing character set "ISO8859-1") I hope this could help tracking down the problem. ================================================= Korean GNOME Community - http://gnome.or.kr
To get rid of those errors you would need to enter a fontset name (I guess a font name that matched multiple fonts, for each encoding), and gnome-terminal needs to provide a way to do so. However, libzvt is broken anyway and doesn't support input methods or double-width chars, so even with that gnome-terminal feature it isn't going to work. libzvt needs to be repaired.
libzvt needs fixing before I can do much. You can try vte instead of zvt if you want, use "vte" module from CVS then build gnome-terminal with "configure --with-widget=vte". It's pretty broken in other ways though. ;-)
*** Bug 82138 has been marked as a duplicate of this bug. ***
Created attachment 8611 [details] [review] temp patch for text input and display
The patch above is by all means temporary, but at least input methods and text display should start working. (I just attached it to ask if I'm doing a right thing or not...) I just did: - add GtkMultiContextIM(based on the patch in #75814) - use XwcDraw to XmbDraw - as the code assumed wchar_t is the same as UTF-32 - this is not right. I'll continue to work on it, to do: - fix cursor location when XFontSet is used - implement preedit-changed callback for XIM BTW, I wonder - is pango support put in libzvt and profterm anytime soon? If so, I'm afraid fixing the libzvt code for the XFontSet support may become duplicate work.
Marking this high to make sun/wipro aware of the issue, but I'm not sure that this is fixable right now so punting out to community 2.0.x. [Maybe 2.2?]
Created attachment 8664 [details] [review] 2nd patch for i18n text input/output and cut & paste
made the 2nd patch - I appreciate if anyone here review it and give me suggestions before I go too far wrong. The patch should make gnome-terminal with libzvt work better in text input and display, as well as cut and paste from/to other apps(or within the gnome-terminal). You have to edit a profile and pick up a proper font for making sure XFontSet can be created for the locale. The 3rd patch will follow shortly - which will do: - preedit change callback implementation - input method selection popup or some other way to select input method.
what env vars. etc. do i need to set to see it working?
setenv LC_ALL or LANG to the name of a multibyte locale. If you have ja_JP.eucJP in a redhat box, then % env LC_ALL=ja_JP.eucJP gnome-terminal For testing text display, run 'date', 'ls -l', or such commands localized for the locale. Make sure to pick up a font to display Japanese characters with XFontSet. (Finding out the font was tough..) In my local Solaris8 box, I tested in en_US, fr, de, ru, ja_JP.eucJP, ja_JP.PCK(Japanese Shift JIS), zh_CN.GBK, and many UTF-8 locales.
As pointed out in bug 75814, this should be a showstopper, so I'm marking up the severity. Also marking 75814 a duplicate of this.
*** Bug 75814 has been marked as a duplicate of this bug. ***
i committed it w/o really testing - we'll see what happens. if more stuff is needed, please reopen. 2002-05-28 jacob berkman <jacob@ximian.com> * configure.in: may as well do 1.116.0 while we're at it * libzvt/libzvt.h: * libzvt/update.c: * libzvt/vt.c: * libzvt/zvtterm.c: i18n patch from Hidetoshi Tajima <hidetoshi.tajima@sun.com> to make i18n work. hopefully it doesn't break everything. fixes #78007
reopening it, as I'm pretty sure there are more to be fixed under this. - preedit changed callback implementation - input method selection menu - PRIMARY selection with multi-byte characters (text should be selected at character boundary rather than byte boundary). - etc...
*** Bug 80311 has been marked as a duplicate of this bug. ***
Hidetoshi: I might suggest that you file separate bugs for each of those issues, and make this bug depend on those. Also, if we've fixed the biggest of the issues, would you agree that is is OK to punt this to 2.0.1 now?
Unfortunately the patch breaks stuff (at least, I think it must be the patch): colors and bold attribute are no longer shown; arrow keys don't work in console apps (e.g. less, vi); and updating is pretty broken (in bash for example Ctrl-D deletes the characters but it's not reflected in the widget, and deleting a line in vi doesn't show the line deleted). Just made a wild guess, and the change to file vt.c is the one causing all these problems. That code doesn't look right, since it's stripping out information from the parsed character thus breaking the state machine. Hidetoshi?
Ahem... it's stripping nothing out :-), but still "information" about the character is lost if it's greater than or equal to 0x80.
ah - if (c < 80) should be if (c < 0x80) in vt.c at least. my fault - I'll commit this right away.
Works great now, at least as far as I tested it. Thanks!
Luis: I think all of the three issues I listed above are important enough to support multibyte locales and they are all to be fixed in 2.0.0 IMO, so I'd rather keep using working on this bugzilla. With a few more weeks, I wish things will be in better shape than now. BTW, how soon will 2.0.1 be released after 2.0.0?
Hidetoshi: if you can promise they'll be fixed in 2.0.0, that's great- but we can't ship a badly broken terminal that doesn't work for anyone in 2.0.0. If that is the case, we'll have to revert the i18n stuff before 2.0.0 is released. The plan for 2.0.1 is 4-8 weeks after 2.0.0, so not that long.
I have filed a new bug, 83528, about weird line spacing and redraw behaviour. Not sure if that is related to this...
Loiue: libzvt is still broken in multi-byte locales but it were more broken before the code change anyway, and in single locales, the code change shouldn't have any impact. If there is, I'd like to look into before reverting the change at all. I'm not sure if #83528 is releted to this, either. Reverting the change will remove the weird line spacing??
A whole lot of changes just came in to what had been a fairly stable zvt, and AFAIK this is the only patch that has gone in. So... I don't know where else to point fingers. Jacob, any thoughts?
i've reverted the changes on HEAD, and have created a branch (libzvt-i18n) where the changes still exist. Hidetoshi, feel free to commit / work on the libzvt-i18n branch.
How's vte nowadays? In terms of I18N features, vte looks much attractive as it already has pango, gtk's input methods and utf8 text handling at least in source code level. I wonder if it's worthwhile to work on zvt i18n - if people don't have much interest in it but plan to move on vte in near future.
this is not the kind of discussion we need to have 7 days before a release!
Well, above all, this kind of bug should not exist 7 days before... Anyway, sorry for the comment above - let me continue to work on libzvt-i18n branch - but I'll be giving up 2.0.0 and targetting 2.0.1 which is 4-8 weeks after. Luis, Jacob - is it okay?
Hidetoshi: yes, I think targeting 2.0.1 would be great. Ideally, though, I'd like to see the patch back in (ideally with fixes for bugs 83400, 83412, 83725, and 83741) ASAP after 2.0.0- if we're going to effect this type of /huge/ change, it needs to be tested as long as possible, not put in at an extremely late date before 2.0.1, so the sooner it gets into CVS after 2.0.0 the better. You may also want to look at/audit the patches posted by Cristiano De Michele to gnome-libs-devel@gnome.org- he may be a bit further along than you on some of these issues and you may benefit from collaboration.
I have a problem in this category with gnome-terminal-1.9.7 and libzvt-1.117.0. gnome-terminal doesn't interpret sticky keys such as 'a -> á and so on. It's the only X application where this happens - xterm and all other gnome applications don't have this problem. I don't have any special locale configured, just the "pt" XkbLayout in /etc/X11/XF86Config-4. I also wanted to note that in a previous gnome-terminal version (a week ago) I had I also experienced problem displaying these kind of characters, but as I update my Gentoo system quite often, that problem got away with an update before I could fill a bug.
Has it been tested that bug 83741 works with the changes reverted?
*** Bug 84838 has been marked as a duplicate of this bug. ***
*** Bug 85960 has been marked as a duplicate of this bug. ***
*** Bug 85469 has been marked as a duplicate of this bug. ***
*** Bug 85986 has been marked as a duplicate of this bug. ***
commit to libzvt-i18n branch for fixing i18n stuff: pango, input methods, selection, and Unicode line buffer handing. Remaining: - Font selection in preference needs to change for selecting pango fonts instead of x_font. - preedit draw should wrap to next line - handling irregal characters
Hidetoshi: Great! thanks. I'll see what I can do about getting this tested.
*** Bug 86134 has been marked as a duplicate of this bug. ***
Yeah, the branch makes i18n pasting work. Like pointed above the font stuff is weird, it uses a variable width default font for me, and obviously the preferences font selector doesnt help until the pango font thing mentioned above gets fixed. I love the antialiasing though :-)
the branch added AA? Eh? Um, not that I'm against AA or anything, but shouldn't that be a separate patch/issue/etc? Tuomas: any stability issues? Jacob: any word on getting the branch into snaps for testing purposes?
there's no word - nobody said i should use this branch. should i?
Oops, yeah, I meant to send you an email about this last week. If you could, please make the snaps build from this branch and send gnome2-snaps an email about it when you do. Thanks.
luis: what's gnome2-snaps email?
hidetoshi: gnome-2-snapshots@ximian.com. I'd suggest letting Jacob handle the announce once it is in the builds.
ok, linux tinderboxen are switched over after i merged stuff down from HEAD. from looking at the diff, the SUN_I18N stuff is going to have to come out before it gets merged to HEAD, and i am not sure that we want to break bin compat (but didn't really check if it was truly broken or not) so we'll see what breaks...
Can everyone here test the patch in it's current form, esp. those of you who reported the original crasher bugs? There is apparently also a report that fonts cannot be set now using this patch; please test that. Thanks! [And thanks, hidetoshi, for taking this one on.]
Ok. I've checked out the libzvt-i18n branch, built it and rebuilt gnome-terminal too. Here is the things that I could gather so far: - a strange different font appears and the font selection dialog doesn't seem to have any effect - typing, copying and pasting "су..." characters works without problems - bold doesn't work - underline is actually over "overline", i.e., the line is draw above the characters and not below - at least one ncurses application isn't displayed correctly (giFTcurs - I'll provide a screenshot shortly) - but other work without problems so far (mutt, vim, lynx, w3m) - gnome-terminal appears to be more slow rendering (I can notice the screen terminal window be updated when scroolling on mutt, or w3m) - gnome-terminal crashes frequently when pressing Control-C while scrolling fast (e.g. doing 'ls -lR'). Here is the backtrace: Backtrace was generated from '/usr/bin/gnome-terminal' 0x40c02639 in wait4 () from /lib/libc.so.6 #0 0x40c02639 in wait4 () from /lib/libc.so.6 #1 0x40c867f8 in sys_sigabbrev () from /lib/libc.so.6 #2 0x40ae40f3 in waitpid () from /lib/libpthread.so.0 #3 0x401c1545 in libgnomeui_segv_handle (signum=-512) at gnome-ui-init.c:620 #4 0x40ae1e3b in raise () from /lib/libpthread.so.0 #5 0x40b8b778 in sigaction () from /lib/libc.so.6 #6 0x40bd2ce1 in free () from /lib/libc.so.6 #7 0x40b1bc64 in g_free (mem=0xfffffe00) at gmem.c:187 #8 0x4007e033 in zvt_term_readdata (source=0x81aaf18, condition=G_IO_IN, data=0x80ee340) at zvtterm.c:3322 #9 0x40b3b58f in g_io_unix_dispatch (source=0x81aaf60, callback=0x4007dd40 <zvt_term_readdata>, user_data=0xfffffe00) at giounix.c:158 #10 0x40b16e95 in g_main_dispatch (context=0x8097f18) at gmain.c:1617 #11 0x40b1770b in g_main_context_iterate (context=0x8097f18, block=1, dispatch=1, self=0x809c4b8) at gmain.c:2161 #12 0x40b135bf in g_main_loop_run (loop=0x8187a68) at gmain.c:2462 #13 0x404b795e in gtk_main () at gtkmain.c:936 #14 0x0805806b in main (argc=1, argv=0xbffff9c4) at terminal.c:1167 #15 0x40b79142 in __libc_start_main () from /lib/libc.so.6 ...and this is it so far. Although this comment is full of complaints, I must say I'm pretty happy for this issue being addressed, and express my gratitude for the excelent work that you all have been doing! Thanks.
Created attachment 9842 [details] giFTcurs screenshot
Just another comment about the giFTcurs thing: the problem seems to be with the upper-right-corner character: +---- | | | or something that is sent at the beginning of each window, because if I force the refresh of the areas that weren't draw (by moving a window over that area), all characters appear correctly from behind, except the above character which stays with the background color. I hope this helps. If anybody wants a further snapshots showing this just say it.
Yeah. Fonts are completely broken, AFAICT. That's Not Cool. Any idea what the problem is there, Hidetoshi?
Yes, fonts are broken. libzvt has to get font preference for pango font like "monospace 14", instead of x-font like "fixed". I think it needs change in gnome-terminal code change for profile. But...
*** Bug 89034 has been marked as a duplicate of this bug. ***
I've updated gnome2 to 20020724... developer snapshot, and now gnome-terminal selects pango fonts properly (I think). The bad thing is I can't see other xfs fonts available (eg. additional truetype fonts), just pango fonts. Maybe, I'm missing some pango config (?) No matter the font I choose, the space between lines is quite ugly.
20020724 now uses vte, since the zvt font fixes were not forthcoming.
I've commited the Zvt side code change for font selection. Together with the profterm side change, which I'll attach after this comment, gnome-terminal can select Pango fonts instead of x fonts on its profile. What's the next step? Can someone help code review to merge libzvt-i18n stuffs into the stable branch and the HEAD?
Created attachment 10115 [details] [review] Patch of profterm to use pango fonts in Zvt
i'd rather have it get some user testing before bothering with code review, as there's no point in reviewing code that doesn't work. however i don't really feel comfortable dumping it on all of our snapshot users. i can spin some packages and let people use them if they want. you might also want to post to some mailing lists asking if people can try out your branch (gnome-i18n *might* be a decent list)
I'll post email to gnome-i18n, gtk-i18n and desktop-devel-list to call for testers.
*** Bug 90405 has been marked as a duplicate of this bug. ***
I18n input methods, font selection and display are all working on Today's libzvt-i18n branch - with id=10115 patch applied to profterm of the HEAD or of gnome-2-0 branch. Marking this "NEEDINFO" for now - and will mark "FIXED" it in a week from today if there is no objection.
For the record, Mandrake cooker has switched to i18n branch of zvt + the profterm patch and there is still some issues (all using latin1 locales such as fr_FR or en) : -bad drawing when the last line of the terminal is scrolled up (one or two pixels on the bottom of the line are not scrolled up and the rest of the line scrolled is truncated..) -when using Monospace font class, even when you change the font size, font glyph size is not changed, only the space around the glyph is changed..
*** Bug 90557 has been marked as a duplicate of this bug. ***
fcrozat: can you document/file all the issues your users are seeing with that branch? it would be much appreciated. In the meantime, I'm going to reopen on fredric's word, but... obviously, we'll close if he doesn't come up with some actual reported bugs :)
Done, I've added bugs #90816, #90818 and #90820 as dependencies of this bug.
*** Bug 85052 has been marked as a duplicate of this bug. ***
commit fix for 90816 and 90818. * libzvt/zvti18n.c (zvt_term_get_xfontset_internal): get font ascent from XFontSet instead of pango metrics(Fixed #90816, #90818).
*** Bug 75487 has been marked as a duplicate of this bug. ***
I forgot where I asked this question last time, so will ask here again. I'd like to merge the libzvt-i18n changes to the HEAD. If any objection, please comment by end of the week.
I have tested a lot the i18n branch. Basic functions of i18n branch are working - include clipboard. Only rejects clipboard, if it contains characters outside terminal's charset. But I am not able properly select my favorite font. It autochooses different font, when I am using ISO-8859-2 resp. UTF-8 and it doesn't use bold font. (tested 2002-10-09)
moving i18n related bugs to internationalization component. This bug will be resolved by merging libzvt-i18n to the HEAD, and applying a gnome-terminal patch to enable pango fonts for Zvt.
*** Bug 87064 has been marked as a duplicate of this bug. ***
*** Bug 97109 has been marked as a duplicate of this bug. ***
Created attachment 12292 [details] [review] i18n patch merged into the HEAD
merged libzvt-i18n to the HEAD.