GNOME Bugzilla – Bug 106618
CJK 'fixed width' font and 's p a c e d o u t' issue
Last modified: 2007-02-05 11:22:43 UTC
Below is the excerpt from Anthony Fok's message to fonts list of XF86: (http://www.mail-archive.com/fonts@xfree86.org/msg01375.html) <quote> I was assigned with the task of dealing with "s p a c e d - o u t " CJK "fixedPitch" font issue in konsole. There are quite a few modern CJK TrueType fonts with the "fixedPitch" flag set to "true" to mean that: * All CJK glyphs have the same "fullwidth" * The ASCII glyphs and other special glyphs have the same "halfwidth" I have submitted a small patch to the FreeType mailing list to deal with the "halfwidth" monospace font issue, and it turns out that Xft has the same issue. Anyhow, to demonstrate the issue, the following screenshot was taken after patching FreeType and before patching Xft: http://anthony.homelinux.net/~foka/xft/halfwidth-rendered-as-fullwidth.png The font therein is a popular "fixedPitch" Chinese TrueType font. After applying the patch to Xft, it looks much nicer: http://anthony.homelinux.net/~foka/xft/halfwidth-even-advance.png </quote> gnome-terminal has a similar problem to that of Konsole when CJK 'fixed width' font is used. When a CJK 'proportional font' (in which all CJK characters are of the equal width but non-CJK characters are of variable width), gnome-terminal makes sorta 'working' guess and works better than Konsole. Still, it's better to support CJK 'fixed width' fonts than using 'proportional fonts'. With the latter, wide characters like 'W', 'M', '@' bump upon neighboring characters or get cut in the middle. </quote> This problem had better be solved upstream in Xft and fontconfig (Anthony has a patch for Xft -see his email above - and more extensive patch is available at http://www.kde.gr.jp/~akito/patch/fcpackage/2_1/) but in the meantime, gnome-terminal can do some heuristics.
Created attachment 14474 [details] a screenshot showing the problem with CJK 'fixed width' font
Created attachment 14475 [details] a screenshot with CJK 'variable-width' font
Please, note that in both screenshots, gnome-terminal keeps CJK characters (East Asian width 'full') *twice* as wide as digits and alphabetic letters (characters with EastAsian width of half. see Unicode Technical Report #11). That's where gnome-terminal is better than KDE Konsole. Unfortunately, while doing so, it assigns the width of 'full width characters' (= *twice* the width of half-width characters) to 'half width characters' and *twice* the width of 'full width characters' to 'full width characters'. As a result, both full width and half width characters are followed by unnecessary extra space of the same width as actual 'ink width'. So, simply removing this factor of '2' for 'fixed width' font would solve the problem. BTW, I guess I was wrong to write that applying either of two patches (to Xft/fontconfig) mentioned in my report would solve gnome-terminal's problem. It seems that because gnome-terminal employs a heuristic to keep the width-ratio (for CJK vs non-CJK,non-combining) 2:1, there'd still be a need for some work to be done on gnome-terminal-side *even after* Xft/fontconfig is fixed.
> gnome-terminal employs a heuristic to keep the > width-ratio (for CJK vs non-CJK,non-combining) 2:1, > there'd still be a need for some work to be done > on gnome-terminal-side *even after* Xft/fontconfig > is fixed. I just confirmed that this is indeed the case. Patching Xft with Anthony's patch solved the problem for vim-gtk2, but gnome-terminal still has the problem.
Created attachment 14635 [details] [review] a patch for pango and xft
With this patch, CJK biwidth(fixed width) fonts work fine with gnome-terminal. Non-CJK fixed width(e.g. Courier) and CJK variable width fonts work just as well as before. This will continue to work even after upstream libraries (Xft and Freetype2) are fixed to better deal with CJK biwidth fonts. FYI, a patch similar to attachment 14635 [details] [review] has been applied to vim-gtk2. BTW, this patch only deals with Xft and Pango 'backends' with X11 unchanged. (GNU unifonts are only bi-width X11 fonts I know of and I haven't tested how they work in gnome-terminal).
Created attachment 14643 [details] a screenshot of gnome-terminal after applying the patch
Looks good, applying.
Thank you for applying my patch.
Recent changes resurrected this bug. There was a couple of 'typo-like' errors in porting the old code. They led to the resurrection. I'm going to upload a patch in a moment.
Created attachment 19757 [details] [review] a new patch against the trunk
Is this problem still there with current CVS?
Created attachment 37471 [details] [review] updated patch I've updated the patch so it applies against current CVS. Please review.
I'm a little unsure about the patch - if the fonts can be of different widths, why only use the last seen? If any of the double-width characters are the same size as the single width characters then we know that we have erroneously based the width on a fixed-width font. Similary, rounding the division up matches the rounding direction of the howmany() macro, but shouldn't we allow a little tolerance? At least I now have a slightly clearer understanding of why that code is there :-)
Created attachment 81695 [details] [review] update of the update Having done that exercise I can see the typo clearly (the missing reset of width before measuring the double_width_chars). So the remaining question does this go in with or without the one pixel slack?
Chris, can you describe how the patch works?
This is my understanding... ;) The current code measures the width of a typical set of ASCII characters, i.e characters that we know should be single-width. The code then measures the width of a few double-width characters. If the two measurements are the same the we know that we are using a fixed-width font and adjust the width corresponding. (modulo the bug) The patch recognises that for any given character we play hunt the font and we should expect the measurements to change corresponding. So every time we detect the font change we compute the current average, perform the fixed-width check, reset the accumulators and continue. This then raises the question of why would a different font have exactly the same width as the first and so should the width comparasion be performed with a tolerance? As it stands the current code has two outright bugs: the failure to reset the width accumulator and the rounding down during the divison. The recheck+reset on font change I believe is better from a mathematical standpoint, but I lack the experience to be able to say whether it actually improves the detection rate or not.
Thanks Chris for the explanation. I don't understand why this is happening in the first place, but your description makes sense.
Likewise. I accept that the heuristic is there to detect something that happened in the wild and as such it makes no sense to carry a buggy version. Feel free to actually detect the 'fixedPitch' property and fix it properly. r1618: 2007-02-04 Chris Wilson <chris@chris-wilson.co.uk> Bug 106618 – CJK 'fixed width' font and 's p a c e d o u t' issue * src/vtedraw.h: * src/vteglyph.c: (_vte_glyph_cache_set_font_description): * src/vtepango.c: (_vte_pango_set_text_font): * src/vtepangox.c: (_vte_pango_x_set_text_font): * src/vtexft.c: (_vte_xft_set_text_font): Detect a change in font between wide-chars and recheck for a fixed-width estimate.
Comment from xterm fontutils.c: /* * fontconfig and Xft prior to 2.2(?) set the width of half-width * characters identical to that of full-width character in CJK double-width * (bi-width / monospace) font even though the former is half as wide as * the latter. This was fixed sometime before the release of fontconfig * 2.2 in early 2003. See * http://bugzilla.mozilla.org/show_bug.cgi?id=196312 * In the meantime, we have to check both possibilities. */