GNOME Bugzilla – Bug 357790
Rendering problem for malayalam consonant RA (U+0D30)
Last modified: 2009-03-26 10:08:18 UTC
Please describe the problem: The rendering of conjuncts formed with u0D30 is wrong. Also, u0D30 when followed by a postbase form splits the original combination and makes it own conjunct with the last consonant. i.e. Combination e.g. u0d15 + u0d4d + u0d2f cannot be followed by u0d30. Steps to reproduce: 1. 2. 3. Actual results: Expected results: Does this happen every time? Other information:
Created attachment 73419 [details] [review] Patch for indic-ot-class-table Modifies the mlymCharClasses[] to change the class of u0d30 from _pb to _cn. u0d30 doesnt have a post base form. Its rendering is similar to u0d31.
Created attachment 73420 [details] [review] Patch for indic-ot.c Since u0d30 and u0d31 have similar rendering, same function is used for u0d30 as for u0d31.
I have tested the output with both of the above patches applied and they work exactly as needed.
2006-10-02 Behdad Esfahbod <behdad@gnome.org> Bug 357790 – Rendering problem for malayalam consonant RA (U+0D30) * modules/indic/indic-ot-class-tables.c: * modules/indic/indic-ot.c (indic_ot_reorder): Fix.
Created attachment 78956 [details] [review] fix for better malayalam support This patch fixes the rendering problems introduced by a) Rahul Bhalerao's patch (357790) b) 0d4d+0d32 (introduced by 355750) and c) removes unnecessary processing of ZWJ The errors are introduced because of fixing rendering engine against bugs in font (lohit_ml used by Fedora Core 6)
The malayalam fix that Rahul Bhalerao has made for the pango module has made the situation that was bad into worse by specifying 2 different specific conditions for three specific charaters (0d4d+0d30/0d31+0d15/0d39). Please note that the rendering that was working fine was broken by incorporating new rules that the language does not have. First of all 0x0d31 is only a consonant. And 0x0d30 is a post base so do not think that both are same because your patch that does so breaks the basic language. Please note that 0x0d4d and 0x0d30 can combine but 0x0d4d and 0x0d31 cannot as per the existing encoding schemes because allowing this would mean two encoding for the same representation resulting in security and other issues. http://www.flickr.com/photos/pravi/335452926/ shows the screenshot before and after the patch (using popular Rachana font). -- Patch by Suresh P ( http://suruma.sarovar.org ) and verified by Praveen A, Swathanthra Malayalam Computing ( http://savannah.nongnu.org/projects/smc )
Duplicated this bug at http://bugzilla.gnome.org/show_bug.cgi?id=397901
Hi, I'm reopening this bug and merging it with 397901 since Praveen wanted to discuss patches in this bug which were applied to trunk but are incomplete or broken. Bye,
*** Bug 397901 has been marked as a duplicate of this bug. ***
So, Praveen, comparing the patch you sent to the Debian BTS and the 1.4.10 release, I see the following chunks were not applied upstream: in modules/indic/indic-ot-class-tables.c: -#define MLYM_SCRIPT_FLAGS (SF_MPRE_FIXUP | SF_NO_POST_BASE_LIMIT | SF_PROCESS_ZWJ) +#define MLYM_SCRIPT_FLAGS (SF_MPRE_FIXUP | SF_NO_POST_BASE_LIMIT ) and in pango-1.14.8/modules/indic/indic-ot.c: - /* for the special conjuction of Cons+0x0d4d+0x0d31 or Cons+0x0d4d+0x0d30 of Malayalam */ - if ((baseConsonant - 2 >= 0) && - (chars[baseConsonant - 1] == 0x0d4d) && - ((chars[baseConsonant] == 0x0d31) || - (chars[baseConsonant] == 0x0d30)) && - ((chars[baseConsonant - 2] >= 0x0d15) && - (chars[baseConsonant - 2] <= 0x0d39))) { - swapChars (&output, -1, -3); - - if (mpreFixups) { - if (mpreFixups->fFixupCount > 0) { - mpreFixups->fFixupCount--; - } - } - } - Could you confirm these are still needed? Is any other change needed in the 1.14 branch (I know we are at 1.16 now, but I didn't have time to check this branch yet).
The first one is to display ZWJ character or not, if I understand correctly (Suresh might be able to explain it) in that case it is safe to remove SF_PROCESS_ZWJ. The second one we don't need anyway (It is an exception which is not required in original script as it is handled by a glyph substitution). I believe the difference is due to additional changes in the upstream (1.15.2) which were not present in debian's version (1.14.8) Currently there are other rendering issues which is fixed by suresh (suruma.sarovar.org) but which would be incompatible with Microsoft's uniscribe rendering engine and so the small changes needs to be made in fonts. We are working on a consensus so that we can submit that patch upstream.
How to add smc-discuss@googlegroups.com to the cc list? Can someone do it?
It seems I failed reopening the bug; really reopening the bug now.
Created attachment 84150 [details] [review] Remainder of the diff with 1.6.0
Praveen said: "Currently there are other rendering issues which is fixed by suresh (suruma.sarovar.org) but which would be incompatible with Microsoft's uniscribe rendering engine and so the small changes needs to be made in fonts." Praveen that move is dangerous. This will make many Malayalam fonts unusable in Linux. Also, users will have to learn this extra fact. As far as possible, Malayalam fonts for MS Windows should work in Linux.
It would not be a hard thing to change all Free fonts to this way (and Suresh has already made Rachana and Anjali to work this way) and as of now the important thing is to fix all the rendering issues and suresh's approach is simple and effective and fixes most of the issues (I will file another bug for the remaining two issues). The uniscribe way of designing a font unnecessarily complicates the rendering engine and it is tough to get it right. The uniscribe engine was designed with all Indian languages in mind and if we move away from that for Malayalam it will solve most of the current rendering issues, as there is no exceptions in Malayalam original script and glyph substitution feature in otf is enough for rendering all complex conjuncts. And if Microsoft want these to work with Windows they can patch the existing releases and select this for their next releases. It is a worry for Microsoft if they want these to work with their systems, not ours (our concern is to have proper rendering in Malayalam and Microsoft's complicated implementation should not be a problem). It is also about the Freedom to decide on our own, how we do computing for our language. The concern about confusion for users is not an issue as all the GNU/Linux distributions package fonts and normally there is a need to hunt for a font in a GNU/Linux distribution. We can make sure that every Free font has aversion that supports this rendering and every distribution ships the correct fonts. If you still want to keep the uniscribe compatibility then it is up to you to drive and fix the issues (which we have already solved in a elegant and yet simple way).
(In reply to comment #11) > The first one is to display ZWJ character or not, if I understand correctly > (Suresh might be able to explain it) in that case it is safe to remove > SF_PROCESS_ZWJ. > > The second one we don't need anyway (It is an exception which is not required > in original script as it is handled by a glyph substitution). > > I believe the difference is due to additional changes in the upstream > (1.15.2) which were not present in debian's version (1.14.8) > > Currently there are other rendering issues which is fixed by suresh > (suruma.sarovar.org) but which would be incompatible with Microsoft's uniscribe > rendering engine and so the small changes needs to be made in fonts. We are > working on a consensus so that we can submit that patch upstream. > The flag SF_PROCESS_ZWJ improves rendering of chillu. The inclusion of ZWJ in gsub table for chillu of the font can solve the issue of unnecessary display of ZWJ after the chillu glyph.
(In reply to comment #15) > Praveen said: "Currently there are other rendering issues which is fixed by > suresh > (suruma.sarovar.org) but which would be incompatible with Microsoft's uniscribe > rendering engine and so the small changes needs to be made in fonts." > > Praveen that move is dangerous. This will make many Malayalam fonts unusable in > Linux. Also, users will have to learn this extra fact. As far as possible, > Malayalam fonts for MS Windows should work in Linux. > cibu, Is it possible to get both forms of conjuncts for zha + va, ie. the one vertical form the and the other horizontal post base form.
(In reply to comment #16) > It would not be a hard thing to change all Free fonts to this way The converse of the statement itself is the problem. One can do this only for the free fonts. Consider somebody has a font which is not free; however, he has the legal rights to use it in his systems. He won't be able to do anything with that font. This will definitely limit the user experience. Consider more specific real life example. Soon all Malayalam newspaper sites would change to Unicode and quite possibly with a font that matches their print edition. A linux user cannot see a newspaper in the desired font, even when the newspaper has allowed their font to be downloaded. This will become a bigger problem once the slow moving dynamic font support is a reality for open systems. The linux users will definitely miss the typography experience. The argument about freedom and simplicity can be said about for not supporting .doc format for OpenOffice. If they hadn't supported .doc format it would have been far less popular product than today. I think user acceptance should be the first priority than design simplicity. > If you still want to keep the uniscribe compatibility then it is up to you to > drive and fix the issues (which we have already solved in a elegant and yet > simple way). This definitely is a valid argument. If we don't have resources, then nothing much can be done about it. I personally does not have much time to devote for this. Cibu
(In reply to comment #18) > Is it possible to get both forms of conjuncts for zha + va, ie. the one > vertical form the and the other horizontal post base form. Yes. zha + virama + va = the vertical stacked form zha + ZWJ + virama + va = horizondal post base form of va Same is true with ya + ra combination also.
Created attachment 84296 [details] ra+virama+kka+o is not rendered correctly Pango rendering of chillu + irattippu + okaram rendering is not correct
Created attachment 84297 [details] ra+virama+kka+o rendered correctly in qt QT rendering this correctly. Incidentally the second case is correct in pango enable Firefox, but not in gedit.
(In reply to comment #19) > (In reply to comment #16) > > It would not be a hard thing to change all Free fonts to this way > > The converse of the statement itself is the problem. One can do this only for > the free fonts. Consider somebody has a font which is not free; however, he has > the legal rights to use it in his systems. He won't be able to do anything with > that font. This will definitely limit the user experience. If people come up with that requirement we can implement those, right now our priority is to to rectify problems with Malayalam rendering, and this is the best solution we have got, anyone is Free to come up with better solution. > Consider more > specific real life example. Soon all Malayalam newspaper sites would change to > Unicode and quite possibly with a font that matches their print edition. A > linux user cannot see a newspaper in the desired font, even when the newspaper > has allowed their font to be downloaded. We can discuss it with the newspaper and have them release their font which Free Software users can use. > > This will become a bigger problem once the slow moving dynamic font support is > a reality for open systems. The linux users will definitely miss the typography > experience. I believe when unicode becomes common dynamic fonts would become irrelevant, the user will be able to see it in any font which he has. The dynamic font is required only when you can't view a page except with its own font, for example you cannot read mathrubhumi with manorama font, but if both uses unicode you don't have to worry about it. > > The argument about freedom and simplicity can be said about for not supporting > .doc format for OpenOffice. If they hadn't supported .doc format it would have > been far less popular product than today. There are people who are willing to support .doc format, no one is against having compatibility with uniscribe, if people are there who are willing to do it, they can come forward. For us the priority is to have perfect rendering for Free Software users. > > I think user acceptance should be the first priority than design simplicity. Why do you think users won't accept a perfect rendering than a crippled one? > > > If you still want to keep the uniscribe compatibility then it is up to you to > > drive and fix the issues (which we have already solved in a elegant and yet > > simple way). > > This definitely is a valid argument. If we don't have resources, then nothing > much can be done about it. I personally does not have much time to devote for > this. We don't have resources to develop uniscribe compatible rendering. In Free Software world users decide what they want with the software, now if users want uniscribe compatibility they can work on it and have it supported. Now those who work on Pango Malayalam support want to fix the rendering issues and they have come up with a solution and I have personally spent time fixing the issues and this is the solution which I propose.
Owen, What do you think of this issue?
(In reply to comment #20) > (In reply to comment #18) > > > Is it possible to get both forms of conjuncts for zha + va, ie. the one > > vertical form the and the other horizontal post base form. > > Yes. > > zha + virama + va = the vertical stacked form > zha + ZWJ + virama + va = horizondal post base form of va > > Same is true with ya + ra combination also. > What is the logic of using ZWJ to get a detached form of a conjunct?
(In reply to comment #25) > (In reply to comment #20) > > (In reply to comment #18) > > > > > Is it possible to get both forms of conjuncts for zha + va, ie. the one > > > vertical form the and the other horizontal post base form. > > > > Yes. > > > > zha + virama + va = the vertical stacked form > > zha + ZWJ + virama + va = horizondal post base form of va > > > > Same is true with ya + ra combination also. > > > > What is the logic of using ZWJ to get a detached form of a conjunct? > It is as per PR 37: http://unicode.org/review/pr-37.pdf
(In reply to comment #26) > (In reply to comment #25) > > (In reply to comment #20) > > > (In reply to comment #18) > > > > > > > Is it possible to get both forms of conjuncts for zha + va, ie. the one > > > > vertical form the and the other horizontal post base form. > > > > > > Yes. > > > > > > zha + virama + va = the vertical stacked form > > > zha + ZWJ + virama + va = horizondal post base form of va > > > > > > Same is true with ya + ra combination also. > > > > > > > What is the logic of using ZWJ to get a detached form of a conjunct? > > > > It is as per PR 37: http://unicode.org/review/pr-37.pdf > I mean the 'logic'.Could you see that. IMHO, ZWNJ is the right choice.
(In reply to comment #27) > (In reply to comment #26) > > (In reply to comment #25) > > > (In reply to comment #20) > > > > (In reply to comment #18) > > > > > > > > > Is it possible to get both forms of conjuncts for zha + va, ie. the one > > > > > vertical form the and the other horizontal post base form. > > > > > > > > Yes. > > > > > > > > zha + virama + va = the vertical stacked form > > > > zha + ZWJ + virama + va = horizondal post base form of va > > > > > > > > Same is true with ya + ra combination also. > > > > > > > > > > What is the logic of using ZWJ to get a detached form of a conjunct? > > > > > > > It is as per PR 37: http://unicode.org/review/pr-37.pdf > > > > I mean the 'logic'.Could you see that. > IMHO, ZWNJ is the right choice. > Well.. This is how it goes: A ZW(N)J is used to define the sub-cluster boundary for which independent shaping decision is made. The decision is as follows: For ZWJ, it will form the conjoining form of the codepoints in the sub-cluster. For ZWNJ, it will avoid forming *any* conjoining form. So Viramas will remain visible viramas. for two possible scenarios sub-clusters are shown as in braces: <(C1, VIRAMA, ZWJ), C2> <C2, (ZWJ, VIRAMA, C2)> This is as per PR 37. BTW, our opinions may not matter that much in these things now. It is a well accepted technical report in UTC. IMHO, ZW(N)J should be avoided as far as possible. Only use should be for U-signs.
What is the summary here?
I thought the patch I submitted reverting the changes by Rahul is committed by Loic, if not please commit it. The discussion went on to touch on other pending issues and we need more time to work on those.
(In reply to comment #30) > I thought the patch I submitted reverting the changes by Rahul is committed by > Loic, if not please commit it. The discussion went on to touch on other pending > issues and we need more time to work on those. Praveen, I did not commit anything in pango directly; I simply compared the patch you sent to the Debian BTS with the current upstream code of pango, and noted a delta. The patch you sent to Debian has been included, but not upstream, and I don't want to resolve this.
2007-05-24 Behdad Esfahbod <behdad@gnome.org> Bug 357790 – Rendering problem for malayalam consonant RA (U+0D30) * modules/indic/indic-ot-class-tables.c: * modules/indic/indic-ot.c (indic_ot_reorder): Commit remaining fix for malayalam.
(In reply to comment #32) > 2007-05-24 Behdad Esfahbod <behdad@gnome.org> > > Bug 357790 – Rendering problem for malayalam consonant RA (U+0D30) > > * modules/indic/indic-ot-class-tables.c: > * modules/indic/indic-ot.c (indic_ot_reorder): > Commit remaining fix for malayalam. Rahul, are you telling me that this change should be reverted?
Ok, forget this. Following up in bug 504810. I have no idea what's correct...
Behdad, We are planning to meet up soon and find a solution that is acceptable to everyone. So if you see a patch supported by me and Rahul (both sides of the argument), that means we have a consensus, just commit it. Please don't make any other changes in Malayalam.
Behdad, Even I have lost a link where the discussion on this bug went on. So it was really nice that you followed up on bug 504810. To consider the real problem in this bug from scratch, i.e. about the character 0D30, lets see the Appendix B in the OpenType specifications for Indic: http://www.microsoft.com/typography/otfntdev/indicot/appen.aspx This character 'Ra' in Malayalam Reformed (MLR) has been marked as post base (p) with a footnote '* Will be reordered at syllable start'. (I am not sure what exact reordering is expected here) Yet, I tried to do it in my initial patch, and it worked for reformed script with few manipulations in font. Apparently, this is not how uniscribe handles this. Fonts developed for uniscribe are found to use 'contextual chain subs' for this case. These fonts do not work on pango. Do you think there is something missing in pango about contextual chain subs that prevents this? Should we try to follow up with the uniscribe approach or should we do the reordering in our own way? Let me know if you need further inputs from me about this ligature and the conflict around it about the reformed and old malayalam script.
I am reopening this bug since the issue about U+0D30 is still unresolved.
Rahul, We have a solution to this. See Suresh's comment. https://bugzilla.redhat.com/show_bug.cgi?id=242016#c46 It needs some effort, but it is worth the effort because Ya or Va cannot form post-base forms in many cases as shown by Suresh. It is worth going the 'akhn' route because it will fix the issue without the need to use complex reordering support from pango or break compatibility with uniscribe (we would still break uniscribe in XP but we have compatibility with what Microsoft believes is the future/improved version of uniscribe) - as you suggested earlier to reverse the xRa sequence in font. Also MalOtf (a popular typewriter/MLR font) already uses it.
Praveen, I would request you not to mix up the 'Ya' and 'Va' related issues in this bug. This bug was filed explicitly for issues related to 'Ra'. (though the discussions went on in other directions earlier) There are other bugs for Ya and Va that are already filed: http://bugzilla.gnome.org/show_bug.cgi?id=470944 and http://bugzilla.gnome.org/show_bug.cgi?id=481198 I hope you understand how difficult it would be for a non-indic developer to keep track of such complex issues :)
Rahul, these two are related (why Ya or Va is better to be 'akhn' than be post-base). If you accept to add some more glyphs to the MLR font for xRa (consonant+virama+ra) conjuncts we have a solution here. First, typewriter reform was never meant for computers it was meant for TYPEWRITERS which cannot have all conjuncts in their keyboard. I failed to understand why you are so stubborn with reducing glyphs, that is never an aim of the MLR script, because there is no restriction in number of conjuncts a font can have. Important point is fixing the problem not reducing glyphs in fonts. And it has been broken for quite a long time and it makes me sad when you block a solution just because you need to add some more glyphs to the font (which has some advantages and also it is already done in other MLR fonts). If you are so much inclined to fix it that way, you have to commit to fix it, since you are blocking a fix for a critical bug.
Pravin, You would have better elaborated the relation between 'Ya', 'Va' issues with 'Ra' issue. About 'Ra', rather than number of glyphs, try to understand the automation hint in my earlier comment. If you incorporate the font rule (GSUB) modifications suggestion made by me earlier this problem would have been fixed. As far the problem with 'Ra' is concerned, (in spite of my doubt about the compatibility in Comment #36,) Lohit Malayalam (font in fedora) and Kartika (font for uniscribe) work seamlessly between uniscribe and pango-1.14.4 i.e. the version of pango that included fixes by LingNing for 'Ra'(0D30) [although I would take back our stand on Rra(0D31) that was also included by then]. Even though that was a well proven solution for both old and new scripts, I am still open enough to review it for any other better solutions if exist. The screenshots following will explain better. [P.S. Only thing I am being stubborn about is the technical correctness of layout engine and fonts. If you do not understand that, then I am helpless about it :( ]
Created attachment 104909 [details] Lohit and Kartika font on pango-1.14.4 Both Lohit (pango) and Kartika (uniscribe) fonts are seen working correctly for the combinations of 'Ra', 'Ya' and 'Va'.
Created attachment 104910 [details] Lohit and Kartika font on Uniscribe Both Lohit (pango) and Kartika (uniscribe) fonts are seen working correctly on uniscribe for the combinations of 'Ya', 'Ra' and 'La'. (The dotted circle in first ligature is not a rendering mistake, its intentional.) I would have given more test results if I had enough efficiency with the input methods on that OS :)
Rahul, With Hiran's fixes for Samyak and Lohit this issue is solved. Can we close this bug now?
I think we could close this bug for now and work independently for Uniscribe compatibility.
All the problems discussed in the bug are now fixed.