After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 106618 - CJK 'fixed width' font and 's p a c e d o u t' issue
CJK 'fixed width' font and 's p a c e d o u t' issue
Status: RESOLVED FIXED
Product: vte
Classification: Core
Component: general
unspecified
Other Linux
: High normal
: ---
Assigned To: Chris Wilson
VTE Maintainers
Depends on:
Blocks:
 
 
Reported: 2003-02-20 17:11 UTC by Jungshik Shin
Modified: 2007-02-05 11:22 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
a screenshot showing the problem with CJK 'fixed width' font (14.71 KB, image/png)
2003-02-20 17:13 UTC, Jungshik Shin
  Details
a screenshot with CJK 'variable-width' font (12.14 KB, image/png)
2003-02-20 17:14 UTC, Jungshik Shin
  Details
a patch for pango and xft (2.84 KB, patch)
2003-02-26 13:17 UTC, Jungshik Shin
committed Details | Review
a screenshot of gnome-terminal after applying the patch (54.56 KB, image/png)
2003-02-26 23:10 UTC, Jungshik Shin
  Details
a new patch against the trunk (7.90 KB, patch)
2003-09-05 02:56 UTC, Jungshik Shin
none Details | Review
updated patch (5.69 KB, patch)
2005-02-14 20:28 UTC, Kjartan Maraas
none Details | Review
update of the update (5.18 KB, patch)
2007-02-01 20:32 UTC, Chris Wilson
none Details | Review

Description Jungshik Shin 2003-02-20 17:11:34 UTC
Below is the excerpt from Anthony Fok's message to fonts list of XF86:
(http://www.mail-archive.com/fonts@xfree86.org/msg01375.html)

<quote>
I was assigned with the task of dealing with "s p a c e d - o u t " CJK
"fixedPitch" font issue in konsole.  There are quite a few modern CJK
TrueType fonts with the "fixedPitch" flag set to "true" to mean that:

   * All CJK glyphs have the same "fullwidth"
   * The ASCII glyphs and other special glyphs have the same "halfwidth"

I have submitted a small patch to the FreeType mailing list to deal with the
"halfwidth" monospace font issue, and it turns out that Xft has the same
issue. 
Anyhow, to demonstrate the issue, the following screenshot was taken after
patching FreeType and before patching Xft:

    http://anthony.homelinux.net/~foka/xft/halfwidth-rendered-as-fullwidth.png

The font therein is a popular "fixedPitch" Chinese TrueType font.

After applying the patch to Xft, it looks much nicer:

    http://anthony.homelinux.net/~foka/xft/halfwidth-even-advance.png
</quote>

gnome-terminal has a similar problem to that of Konsole when CJK 'fixed
width' font is used. When a CJK 'proportional font' (in which
all CJK characters are of the equal width but non-CJK characters
are of variable width), gnome-terminal makes sorta 'working' guess and
works better than Konsole. Still, it's better to support
CJK 'fixed width' fonts than using 'proportional fonts'. With the latter,
wide characters like 'W', 'M', '@' bump upon neighboring characters
or get cut in the middle. 


</quote>


This problem had better be solved upstream in Xft and fontconfig
(Anthony has a patch for Xft -see his email above -
and more extensive patch
is available at http://www.kde.gr.jp/~akito/patch/fcpackage/2_1/)
but in the meantime, gnome-terminal can do some heuristics.
Comment 1 Jungshik Shin 2003-02-20 17:13:14 UTC
Created attachment 14474 [details]
a screenshot showing the problem with CJK 'fixed width' font
Comment 2 Jungshik Shin 2003-02-20 17:14:45 UTC
Created attachment 14475 [details]
a screenshot with CJK 'variable-width' font
Comment 3 Jungshik Shin 2003-02-20 17:27:12 UTC
Please, note that in both screenshots, gnome-terminal keeps CJK characters
(East Asian width 'full') *twice* as wide as digits and alphabetic
letters (characters
with EastAsian width of half. see Unicode Technical Report #11).
That's where gnome-terminal is better than KDE Konsole. 
Unfortunately, while doing so, it assigns the width of 'full width
characters'
(= *twice* the width of half-width characters)
to 'half width characters' and *twice* the width of 'full width
characters'
to 'full width characters'. As a result, both full width and half width
characters are followed by unnecessary extra space of the same width
as actual 'ink width'. 

So, simply removing this factor of '2' for 'fixed width' font would 
solve the problem.  

BTW, I guess I was wrong to write that applying either of two
patches (to Xft/fontconfig) mentioned in my report would solve
gnome-terminal's problem. It seems that because 
gnome-terminal employs a heuristic to keep the 
width-ratio (for CJK vs non-CJK,non-combining) 2:1, 
there'd still be a need for some work to be done
on gnome-terminal-side *even after* Xft/fontconfig
is fixed. 
 
 
Comment 4 Jungshik Shin 2003-02-21 14:48:45 UTC
> gnome-terminal employs a heuristic to keep the 
> width-ratio (for CJK vs non-CJK,non-combining) 2:1, 
> there'd still be a need for some work to be done
> on gnome-terminal-side *even after* Xft/fontconfig
> is fixed. 

  I just confirmed that this is indeed the case.
Patching Xft with Anthony's patch solved the problem
for vim-gtk2, but gnome-terminal still has the problem.
 
Comment 5 Jungshik Shin 2003-02-26 13:17:47 UTC
Created attachment 14635 [details] [review]
a patch for pango and xft
Comment 6 Jungshik Shin 2003-02-26 13:24:14 UTC
With this patch, CJK biwidth(fixed width) fonts work fine with
gnome-terminal.
Non-CJK fixed width(e.g. Courier) and CJK variable width fonts work
just as well as before.
This will continue to work even after upstream libraries (Xft and
Freetype2)
are fixed to better deal with CJK biwidth fonts.
 
FYI, a patch similar to attachment 14635 [details] [review] has been applied to vim-gtk2. 

BTW, this patch only deals with Xft and Pango 'backends' with X11 
unchanged. (GNU unifonts are only bi-width X11 fonts I know of and I
haven't tested how they work in gnome-terminal). 

Comment 7 Jungshik Shin 2003-02-26 23:10:19 UTC
Created attachment 14643 [details]
a screenshot of gnome-terminal after applying the patch
Comment 8 Nalin Dahyabhai 2003-02-26 23:42:09 UTC
Looks good, applying.
Comment 9 Jungshik Shin 2003-03-07 13:21:38 UTC
Thank you for applying my patch. 
Comment 10 Jungshik Shin 2003-09-05 02:55:38 UTC
Recent changes resurrected this bug. There was a couple of 'typo-like'
errors in porting the old code.  They led to the resurrection. I'm
going to upload a patch in a moment. 
Comment 11 Jungshik Shin 2003-09-05 02:56:38 UTC
Created attachment 19757 [details] [review]
a new patch against the trunk
Comment 12 Kjartan Maraas 2004-10-18 09:34:36 UTC
Is this problem still there with current CVS?
Comment 13 Kjartan Maraas 2005-02-14 20:28:42 UTC
Created attachment 37471 [details] [review]
updated patch

I've updated the patch so it applies against current CVS. Please review.
Comment 14 Chris Wilson 2007-02-01 20:12:46 UTC
I'm a little unsure about the patch - if the fonts can be of different widths, why  only use the last seen? If any of the double-width characters are the same size as the single width characters then we know that we have erroneously based the width on a fixed-width font. Similary, rounding the division up matches the rounding direction of the howmany() macro, but shouldn't we allow a little tolerance?
At least I now have a slightly clearer understanding of why that code is there :-)
Comment 15 Chris Wilson 2007-02-01 20:32:23 UTC
Created attachment 81695 [details] [review]
update of the update

Having done that exercise I can see the typo clearly (the missing reset of width before measuring the double_width_chars). So the remaining question does this go in with or without the one pixel slack?
Comment 16 Behdad Esfahbod 2007-02-03 18:13:45 UTC
Chris, can you describe how the patch works?
Comment 17 Chris Wilson 2007-02-03 18:50:41 UTC
This is my understanding... ;)

The current code measures the width of a typical set of ASCII characters, i.e characters that we know should be single-width.
The code then measures the width of a few double-width characters. If the two measurements are the same the we know that we are using a fixed-width font and adjust the width corresponding. (modulo the bug)

The patch recognises that for any given character we play hunt the font and we should expect the measurements to change corresponding. So every time we detect the font change we compute the current average, perform the fixed-width check, reset the accumulators and continue.

This then raises the question of why would a different font have exactly the same width as the first and so should the width comparasion be performed with a tolerance?

As it stands the current code has two outright bugs: the failure to reset the width accumulator and the rounding down during the divison.

The recheck+reset on font change I believe is better from a mathematical standpoint, but I lack the experience to be able to say whether it actually improves the detection rate or not.
Comment 18 Behdad Esfahbod 2007-02-03 19:50:26 UTC
Thanks Chris for the explanation.  I don't understand why this is happening in the first place, but your description makes sense.
Comment 19 Chris Wilson 2007-02-04 22:01:24 UTC
Likewise. I accept that the heuristic is there to detect something that happened in the wild and as such it makes no sense to carry a buggy version. Feel free to actually detect the 'fixedPitch' property and fix it properly.


r1618: 2007-02-04  Chris Wilson  <chris@chris-wilson.co.uk>

	Bug 106618 – CJK 'fixed width' font and 's p a c e d o u t' issue

	* src/vtedraw.h:
	* src/vteglyph.c: (_vte_glyph_cache_set_font_description):
	* src/vtepango.c: (_vte_pango_set_text_font):
	* src/vtepangox.c: (_vte_pango_x_set_text_font):
	* src/vtexft.c: (_vte_xft_set_text_font):
		Detect a change in font between wide-chars and recheck for a
		fixed-width estimate.

Comment 20 Chris Wilson 2007-02-05 11:22:43 UTC
Comment from xterm fontutils.c:
/*
 * fontconfig and Xft prior to 2.2(?) set the width of half-width
 * characters identical to that of full-width character in CJK double-width
 * (bi-width / monospace) font even though the former is half as wide as
 * the latter.  This was fixed sometime before the release of fontconfig
 * 2.2 in early 2003.  See
 *  http://bugzilla.mozilla.org/show_bug.cgi?id=196312
 * In the meantime, we have to check both possibilities.
 */