Created attachment 61358 [details]
DejaVu Sans with overshot kerning and then none

Te overkerned followed by Te not kerned

Created attachment 61359 [details]
YanoneKaffeesatz Te kerned and not kerned

Yanone Kaffeesatz Te kerned folled by Te not kerned.

DejaVu Sans shows this behaviours sometimes too.

Honestly, judging from the sheer amount of bugs with kerning involving DejaVu fonts, I'm 99% sure that it's a bug in the font, not Pango.

That doesn't really matter though.

What was the latest version that this was working with?  We didn't change much in Pango recently.  Not at all things that affect these kind of things.  I'm afraid you are on yourself on debugging this.

Did you happen to install FreeType 2.2 snapshots by any chance?

I have FreeType 2.1.10-1ubuntu1.
I totally understand your concern, this is why I didn't report this earlier.

However these are not the only fonts I've seen this with.
Bitstream Cyberbase
(ftp://ftp.netscape.com/pub/communicator/extras/fonts/windows/) has kerning for
"Te" or "To", however I see none.
jGaramond (http://www.janthor.de/jGaramond/), ditto.
Pigiarniq Regular (http://www.gov.nu.ca/font.htm or
http://www.tiro.com/syllabics/resources/Compiled%20Data%20Sources/Fonts/Pigiarniq/Pigiarniq_win_ttf.zip)
has kerning for "Te" and "To" but shows similar behaviour to Yanone Kaffesatz
("TeTe" only has the first occurence with kerning).

On the other hand Gentium works properly.
Sorry if I can't find more examples, free fonts with kerning are hard to find.

Could it just be my setup that's broken? (testing Ubuntu Dapper Drake)

Comparing with an older version of DejaVu Sans it seems the overkerning is a recent DejaVu fonts bug.

The first kerning followed by no kerning might be a old bug in Pango. 
I've only started looking at kerning recently due to the overkerning in DejaVu Sans. I would have never noticed kerning inconsistancies in other fonts otherwise.

bug reports on Ubuntu's Launchpad
https://launchpad.net/distros/ubuntu/+source/ttf-dejavu/+bug/35470
https://launchpad.net/distros/ubuntu/+source/pango1.0/+bug/34178

DejaVu LGC, which is simply DejaVu with all the Arabic and Armenian characters removed, leaving Latin, Greek, and Cyrillic, looks fine. Why should the presence of other scripts affect the rendering of Latin?

Actually, the kerning of DejaVu LGC Sans is still inconsistent (the same pair of glyphs kerned differently in different places), but it's never as badly overshot.

That's all font issues.  You write bogus OpenType tables, you get misrendered glyphs.  Garbage In, Garbage Out principan in action.

*** Bug 335703 has been marked as a duplicate of this bug. ***

Created attachment 61872 [details]
More extreme

This should say "officials", it is the "ic" part that seems to be the trouble, as I have noticed this bug for a few days now, using Ubuntu 6.06.

I did a bisection search and it looks good with 1.11.0 but bad with 1.11.1. Did Pango start looking at some tables it ignored before?

I leapt to the conclusion that it was a Pango bug because it only appears with recent versions of Pango and not with any other font software, but if there's something wrong with part of our font and only Pango even looks at that part, then that would also explain it.

Yes, with 1.11.1 pango started using the OpenType engine for Latin script too.  Look at the modules/basic/basic-fc.c that was revamped.  Previously it was doing fallback shaping, only applying the TrueType kern table, but now it's doing the recommended tables: GSUB ones: ccmp, liga, clig, and GPOS ones: kern, mark, and mkmk.

So, you are not getting the same behavior with Qt?  We almost share the same OpenType layout code.

Nobody has _noticed_ this problem on Qt, or on other engines like Cleartype, but I can't say for sure.

Most likely nothing else you are testing with pays attention to the
kerning tables.

Created attachment 61914 [details]
Qt4 has no problems with kerning

The problem does not occur with Qt4 (attached screenshot is made from textedit, one of Qt4 demo programs)

Scribus, the only Qt3 I know that has kerning, has no problems

And OpenOffice 2 also has no problems with kerning

And how does the same string using the same font look under Pango?

Created attachment 61916 [details]
screenshot Qt4/Pango with DejaVu Sans and DejaVu LGC Sans

Here's what Qt4 with DejaVu Sans 2.4 looks like compared to Pango.
DejaVu LGC Sans does not overkern with Pango.

Another side effect of having Arabic in DejaVu Sans is anchors not working anymore. Removing all Arabic anchors fixes it. But that might be Fontforge not writing clean tables. I don't know if anchors work on other platforms. "ë̌" <e, U+0308, U+030C> should stack cleanly.

There two issues to be dealt with, the overkerning one apparently DejaVu Sans 2.4 + Pango specific, and the inconsistant kerning.

Note that putting Arabic into DejaVu Sans is a shockingly bad idea,
and I don't think it's worth even thinking about investigating
problems with it. (The main reason it's a horrid idea is that
Arabic fonts and Latin fonts don't have compatible metrics.)

That being said, one problem per bug report please!

Seems Fedora is hit too:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=186662

(also see the discussion on dejavu lists:
http://sourceforge.net/mailarchive/message.php?msg_id=15185495 )

As for Owen's remark in comment #20 :

1. if you mean this is a shockingly bad idea because the tools are not ready and we've been caught with our pants down I can only agree

2. if you mean multi-script fonts should be prohibited or you can ignore them that's another thing altogether. Users want multi-script fonts. There are few now but there were few unicode systems some years ago too. fontconfig substitution is a short-term fix, not a long-term solution (if only because yhe aliasing rules are not the same on other systems and ignored by non-fontconfig apps). People are making Vera derivatives at last (which was the original aim, remember), all sorts of scripts are beind added and then consolidated in dejavu, get over it and please make it work.

I still remember the bad old ASCII days where you could create any dual-document you wanted, as long as one of the langages was english (because the only character overlap was in the latin part). Now that fonts are catching up, let's get rid of the last remnants of this situation.

You mean you do whatever users want you to do?  Ignore the users, design and solve the problems with developers.  We write the software, we know how fonts are handled, not users.  We know the problems of multi-script fonts, not users.  Lets get over it, don't ruin all the beauty of fontconfig, instead fix its bugs.

With current technology, combining scripts with incompatible metrics
simply does not work. You'll either get overlapping lines, or hugely
spaced out lines.

(Greek, Latin, Cyrillic are compatible, probably also Georgian, Armenian,
Ethiopic, etc. Arabic, Thai noteably incompatible. Indic scripts also
incompatible though a little less strongly.)

If you add BASE tables to the font, and adapt every piece of software
in use, you may be able to get it to work. But then you've created a
way of using fonts that isn't compatible with what other platforms do.

fonconfig/Pango do a reasonable job of combining multiple fonts; there
are some issues with sizing and baseline alignment that need work, 
but those are best addressed in the context of multiple fonts; after
all, doing typography with multiple fonts for multiple scripts is
something that needs to work!

If you have someone that wants to draw a Arabic design that coordinates
well with Deja Vu, great!, release it as Deja Vu Arabic. But don't
go down the FreeSans rathole of throwing fonts together in the blind
and unreasoned pursuit of coverage statistics :-)

The only font that I know of that has coverage outside of latin-like
fonts and "works" is Microsoft's Tahoma, and that achieves a uniform
line spacing by using unusual and strange looking variants of scripts
like Thai that are readable but not normal typography in that script.

Owen,
it's not a  blind and unreasoned pursuit of coverage statistics; it's what users want.

The fantasy of dedicated script fonts is just that, a fantasy :
1. All the "dedicated" fonts overlap, especially in the core latin parts so you can not reasonably "combine" them
2. Even if they did not overlap asking the users to manage font collections because of technical problems just does not work. Users are always more stubborn than technologists. They want a single font. Are you ready to embark on a crusade to convince them they're wrong ?

I'm afraid this is something FOSS will have to tackle and solve by itself. As long as major closed software editions are segmented by langage they just do not have the incentive FOSS distributions with their large i18n coverage have.

And that means FOSS font designers releasing fonts with large coverage (as the dejavu people are kindly doing) but also the FOSS GUI infrastructure people figuring out how to display them.

Maybe the solution is to add a new table to fonts which allows managing all these scripts in a single file ? If the (numerous) existing font tables have unsolvable limitations.

We've already created a way of using fonts that isn't compatible with what other platforms do. Just write a document that references a fontconfig alias, and try to display it the same way on another platform (or a fontconfig system with a different aliasing table for that matter). Trying to work like everyone else is good but only as long as everyone else has solved your problems.

If you separate fonts then combine them at fontconfig level you're as incompatible with other platforms as if you relied on a table other platforms can not read (yet)

Today I don't see a good way to do multi-script documents. And in a globalized world, they're more and more numerous.

To answer Behdad.

I won't pretend to know how the software work. But I've been exposed to many many multi-script documents (from nuclear powerplant plans to textile automation brochures). And the #1 problem every time was how software insisted on separating the langages (by font, encoding, whatever) when every single document user wanted a common presentation.

Go in a store. Read the doc that comes with any piece of electronic kit nowadays. I'll be very surprised if most of it is not in multiple languages and if the doc creator didn't expend a lot of efforts to make all langages equal (very bad for business if pakistanies think indic scripts are bigger/better/more favoured than arabic). FreeSans and dejaVu didn't go the same route to get coverage statistics. They went the same route bacause that's the route users ask for.

> The fantasy of dedicated script fonts is just that, a fantasy :

It rather seems to me that you are chasing the fantasy of a working
all-scripts font here. Not a good sign for including deja vu as the
default font in Fedora, I'm afraid.

This is interesting discussion should probably happen on gnome fonts list or the internationalization list instead of here ;).

Irrelevant of whether a many-script font is a good thing or not, it should work with Pango as it does with other font renderers.

Should we open a different bug report for the kerned/not kerned issue since this one seems to focus on DejaVu Sans with Arabic OT tables?

Sorry Denis for helping on hijacking your bug...

Nicolas: No, that is not users want.  Users don't know what fontconfig is, users don't know how font selection is done, users don't know all these things.  What users want is that their software works.  If the simplest way to give them that for you, the font developer, is to stuff every glyphs somebody contributes into a single bin, this bug is the result, let everyone suffer it.

All Microsoft fonts with Arabic glyphs are absolutely ugly for Persian, the Arabic glyphs in DejaVu will definitely be ugly for Persian too, for a very simple reason:  Arabic fonts are ugly for Persian speakers.  Persian simply prefer other styles.  Persian fonts and Arabic fonts are not suitable for Urdu.  And these three languages share the same script, aka the Arabic script.  So, no matter how much effort you put into making the Arabic glyphs in DejaVu perfect, it will be:

  1) Not needed for 5 billion of people in the world that don't know Arabic, you are just making a lean Latin font, a huge pig now.

  2) Unusable for Persian and Urdu speakers, a good few hundred millions, and they cannot easily use DejaVu with their favorite Persian/Urdu font anymore.

  3) Not necessarily the style they want to use with their DejaVu font for those who actually are Arabic speakers, and again, you have made it harder for them to choose which font to use for which script.

You may want to take a look at fontmanager project that is being developed and at other efforts to solve this problem, instead making a decision based on what end-users say.  When you design new fonts, you should also write a good fontconfig configuration file for them.  I've done that for Persian, and the result is beyond anything you can ever achieve using the single-font approach.

Moreover, developing a font for a single script is already hard enough.  Instead of adding more and more glyphs, DejaVu developers may want to stop for a while and do some testing on their fonts before pushing them on millions of desktops.    I've been watching the Pango bugs for a few years now, and I don't need to tell you how many bugs have been reported here for say Bitstream Vera Sans, and how many for DejaVu Sans...

Of course, one can fork DejaVu and divide it into multiple fonts anyway.

Does Pango support BASE table?  If it supports it well, how is "different metrics" relevant?

No it doesn't.  Even Pango does, Qt doesn't, and Firefox doesn't.

(In reply to comment #27)
> Sorry Denis for helping on hijacking your bug...

And I'm very sorry, but I have to answer :(

Behdad, all you're saying is something I already know, that is current fonts lack info to compose them sanely. You're proposing to put this info in fontconfig files, or in gconf keys managed by fontmanager, or whatever (and there I'm ROTFL since Owen's first argument was to avoid creating a
way of using fonts that isn't compatible with what other platforms do.)

Your first postulate seems to be fonts as they are today are hopeless, and you have to fix them from the outside. But here you have a font design project which is ready to help you at the font design stage.

Instead of creating a fontmanager that will probably be used to map from scripts to fonts (like the firefox one today which is so complex I defy you to find people which have changed more than 2-3 langages) how about :
1. defining which glyphs needs alternatives (info wolrdwide projects like gnome can collect)
2. put these aletrnative in predefined slots

and just have the whole thing work as-is ?

You'll probably object this will lead to the creation of very fat fonts, but distros will install all the split fonts anyway, and these split fonts will probably overlap in the latin range, so splitting won't win anything except for LFS installs

(In reply to comment #27)

> Moreover, developing a font for a single script is already hard enough. 
> Instead of adding more and more glyphs, DejaVu developers may want to stop for
> a while and do some testing on their fonts before pushing them on millions of
> desktops.    I've been watching the Pango bugs for a few years now, and I don't
> need to tell you how many bugs have been reported here for say Bitstream Vera
> Sans, and how many for DejaVu Sans...

BTW, since Vera is a Gnome project, and Dejavu a Vera fork, would adding a DejaVu component to gnome bugzilla (if DejaVu people agree) where you could reassign bugs make your life easier ?

> Your first postulate seems to be fonts as they are today are hopeless, and you
> have to fix them from the outside. But here you have a font design project
> which is ready to help you at the font design stage.

Yes, we trust you guys to know what you're talking about, so give us some recommendations. We say it's a Pango bug, you say DejaVu is messed up, well, what's wrong with it? How should we fix it? We'd be willing to consider anything as long as it doesn't screw up our rendering on Qt or Windows.

Well, the pangoheads already spelt out the options:

1) split off scripts which are incompatible from a font-metric point of view
   like Arabic

2) work on support for OpenType BASE tables in pango, which can theoretically
   overcome the font metric problems. In this case, you have to live with the
   fact that your font is broken in every piece of software now, and in every
   piece of software except pango after the BASE support is completed and
   released...

About the kerning issue:

Testing with DejaVu shows that the problem is probably caused by the right-to-left anchors needed for Arabic: removing these anchors from DejaVu solves the problem.

Nicolas: All I'm saying is that I don't want anybody to *force* it on me which Persian font family to use with my favorite Latin font family.  That's all.

Keenan: OpenType tables are full of numbers.  One of them is wrong and you see bugs like this.  Qt is not using OpenType tables for Latin I guess.  And did anyone test the offending font&sequence on Windows actually and it did work?  If it didn't, it's definitely a font bug.  If it did, it's either 1) a font bug, hidden by a windows bug, 2) a Pango bug.  In both cases, we don't have any problem to change Pango to render the font correctly. We have had other cases of this, see bug 101079 and bug 170180 for cases where a buggy font was rendering fine on Windows, and not with Pango, because Pango is much more restricted and standard compliant that Windows.  BUT, somebody has got to debug the problem and come up with a conclusion that either Pango or the font is not following the OpenType spec, for this particular reason, here in this place, and here is a fix.  I've done that in the past, and am willing to do if somebody attaches a small test font.  But with a monstrous font with thousands of table entries, I'm afraid it's not me.

The overkerning in DejaVu Sans can be fixed either by removing the Arabic anchors, or by changing the kerning pairs currently present. All the kerning pairs are marked for the "cyrl{dflt} grek{dflt} latn{dflt}", changing that to just "latn{dflt}" fixes overkerning and diacritics stacking. 
Unless it has any secondary effects, it seems to be the best fix since we can keep Arabic anchors.

The inconsistent kerning does remain as for other fonts.

I still think someone should figure out *why* that's happening.

I made a simple font with basic Latin and Arabic. With "Te" kerned for "cyrl grek latn" and "To" kerned with the same value for "latn".
There is only one anchor (rtl) for the base U+0637 and the mark U+64B.
http://home.sus.mcgill.ca/~moyogo/fonts/test/Jaja-Arabic.ttf

If you test it with "Te To طً" you'll see it reproduces the bug, "Te" is overkerned compared to "To" and the diacritic is not placed according to the GPOS table.

I'm happy to understand in this thread why my GNOME desktop look so ugly.

Just my 2 cents in this discussion:
I don't look for fonts that cover ALL Unicode glyphs in one file (it's a bad technical approch), but fonts that render very well in one or more defined Unicode scripts.
I also look for fonts able to render multi-scripts documents in a correct way.

As an end user, I would like to simply choose the family "Vera Sans" in the font selection dialog of Abiword or OOo and the font engine choose for me the more appropriate font {script|file|...} between "Vera Sans LGC", "Vera Sans Arabic", "Vera Sans Persian"... depending of my locale, default language of the document, paragraph, <choose what you want>

As a power user, I would like the same font selection dialog box let me bypass the automatic font substitution by specifying the full font name or render script.

For now, I really prefer fonts that render very well for some defined Unicode scripts, that fonts full of glyphs that render badly in any scripts.
I switched from "Vera Sans" to "Vera Sans LGC" for this reason except for "Vera Sans LGC Mono" that render very badly in gnome-terminal.

It appears to be a problem upstream, fixed here - http://dejavu.sourceforge.net/wiki/index.php/Main_Page

(In reply to comment #12)
> Created an attachment (id=61872) [edit]
> More extreme
> 
> This should say "officials", it is the "ic" part that seems to be the trouble,
> as I have noticed this bug for a few days now, using Ubuntu 6.06.

This is actually looks like a different bug: firefox not getting the advance for ligatures right with justified text when compiled with Pango support.  More information here:
  https://launchpad.net/distros/ubuntu/+source/firefox/+bug/37828
  https://bugzilla.mozilla.org/show_bug.cgi?id=331716

(In reply to comment #39)
> I'm happy to understand in this thread why my GNOME desktop look so ugly.
> 
> Just my 2 cents in this discussion:
> I don't look for fonts that cover ALL Unicode glyphs in one file (it's a bad
> technical approch)

Why? It's not working now, but nothing else is working now either (working as defined by your next §)

> but fonts that render very well in one or more defined
> Unicode scripts.
> I also look for fonts able to render multi-scripts documents in a correct way.

Like everyone else.
I'd also ask for fonts that can be used by everyone, including people lacking a swiss bank account

> As an end user, I would like to simply choose the family "Vera Sans" in the
> font selection dialog of Abiword or OOo and the font engine choose for me the
> more appropriate font {script|file|...} between "Vera Sans LGC", "Vera Sans
> Arabic", "Vera Sans Persian"... depending of my locale, default language of the
> document, paragraph, <choose what you want>

How is it functionnaly different from putting all the glyphs and glyph variations produced by a single team in a single font, and have the rendering engine choose the right glyph variant depending of your locale, default language of the document, paragraph, <choose what you want>

(except that the font setting saved will actually mean something when doc moves from system to system, and not be a virtual alias you share with no one)

> As a power user, I would like the same font selection dialog box let me bypass
> the automatic font substitution by specifying the full font name or render
> script.

Again glyph location has no incidence whatsoever there, since you need to order ranges-of-glyphs-in-fonts not fonts anyway (everyone and its mother is overlapping in some ranges like base latin) 

> For now, I really prefer fonts that render very well for some defined Unicode
> scripts, that fonts full of glyphs that render badly in any scripts.

Which is a compromise you made, and others may have made another one.
Situation would be different if there was a perfect solution, but all I see here is people writing at each other "your compromise suck because of <insert personal situation>" mine is better (and will handle your problems someday probably)

If FOSS font projects (FreeFont, dejaVu) go one way (full coverage, someday the renderer will handle it) and rendering go the other (split everything, someday the font tools will reassemble it) everyone will lose (somehow I don't see Behdad drawing the font families he asks for nor dejaVu rewriting pango).

All that has been established do far is there's a lot of work to do.

I think enough of font discussion here.  Back to the bug.

Just so this is written down somewhere. 

The overkerning occurs with fonts that have kern pairs or classes set in more than one script. For example DejaVu Sans 2.4 had its kern pairs in "latn(dflt),cyrl(dflt),(grek(dflt)". Fonts like Calibri (from Vista Beta) will display the same behavior in Pango. Pango applies the kerning for the pair for each script, instead of just one.

The weird part is that DejaVu Serif has a similar definition for kern pair but Pango only applies it once.
Maybe Calibri and DejaVu Sans have bad tables but other font shapers seem to do fine. It could be argued that fonts should not do this or do that, but what should we do, have a more flexible library or fix each font (including professional ones) that's not to its liking?

Ok, I hereby declare that a Pango bug :).

Here is what's happening in the OpenType basic module:

    for (i = 0; i < G_N_ELEMENTS (scripts); i++)
      {
         PangoOTTag script_tag = FT_MAKE_TAG (scripts[i][0], scripts[i][1], scripts[i][2], scripts[i][3]);
         guint script_index;


         if (pango_ot_info_find_script (info, PANGO_OT_TABLE_GPOS, script_tag, &script_index))
           for (j = 0; j < G_N_ELEMENTS (gpos_features); j++)
             {
               PangoOTTag feature_tag = FT_MAKE_TAG (gpos_features[j][0], gpos_features[j][1],
                                                     gpos_features[j][2], gpos_features[j][3]);
               guint feature_index;

               /* 0xffff means default language */
               if (pango_ot_info_find_feature (info, PANGO_OT_TABLE_GPOS, feature_tag, script_index, 0xffff,&feature_index))
               {
                 pango_ot_ruleset_add_feature (ruleset, PANGO_OT_TABLE_GPOS, feature_index, 0xffff);
               }
             }

So we simply add every feature for every script to the ruleset...

The solution is the fc-modules revamp that Denis and I have been talking about.  Ideally, I think we should be keeping one ruleset per script-language pair.

I am currently Pango 1.14.5 and DejaVu 2.7 on Ubuntu 6.10. If I compare with the screenshot in attachment #6 [details], I don't see any kerning errors. At this moment, is this bug still in existence?

This bug is 8-5 months old and the last comment said it is a Pango bug, but this bug is still marked as UNCONFIRMED?

Could it be that this bug is in fact already fixed, but that the status of this bug has not been updated (that would explain why I don't see problems in the font kerning at this moment)?

(In reply to comment #46)
> I am currently Pango 1.14.5 and DejaVu 2.7 on Ubuntu 6.10. If I compare with
> the screenshot in attachment #6 [details] [edit], I don't see any kerning errors. At this
> moment, is this bug still in existence?

Yes, it is.  But DejaVu has worked around it.

> This bug is 8-5 months old and the last comment said it is a Pango bug, but
> this bug is still marked as UNCONFIRMED?

We don't care much about the UNCONFIRMED status.

> Could it be that this bug is in fact already fixed, but that the status of this
> bug has not been updated (that would explain why I don't see problems in the
> font kerning at this moment)?

No.

The bug occurs when a kerning pair (e.g. "Te") is contained simultaneously in several GPOS tables, e.g. latn{dflt}, cyrl{dflt} etc. Pango accounts this distance multiple times probably summing. Obviously it should do it once picking a value from the suitable table, not the all.

*** Bug 440975 has been marked as a duplicate of this bug. ***

This is fixed btw.