GNOME Bugzilla – Bug 321490
arabic, hebrew: character alignment not working properly
Last modified: 2019-01-29 11:17:11 UTC
Arabic writing direction is not correct. Actual behavior: from left to right. Expected behavior: from right to left.
Created attachment 54765 [details] Snapshot for gnome-terminal The snapshot shows I input 'a', 'b', 'c', 'd', 'e'.
This WONTFIX in vte. Not in my sight. You can run BiCon inside vte to get a similar behavior: http://www.arabeyes.org/project.php?proj=BiCon
Created attachment 111119 [details] [review] pacth to vte-0.16.13 which enables BiDi and shaping this patch will disable xft2 because I added the support to pango and pangox backends only
Salam, I fix and it works apply this patch to version vte-0.16.13 cd vte-0.16.13 patch -p1 < vte-0.16.13-2.patch and then rebuild it and it will work gnome-terminal and rhgb will work as a result my patch does the following 1. disable xft2 (till similar procedure is done to it) 2. make pango backend handles request line by line if possible not char by char yes this is a primitive support like the one in konsole of kde 3.x there are some known problem like when you are typing because only one char gets updated the problem is due to that fact vte does not update every thing it only updates the changes but if there is a trick to add a timeout the update the entire screen every second it will be good (I don't know how, because I don't know the logic of vte)
Created attachment 114595 [details] [review] fix bidi problems (apply with patch -p1) better than the previous patch
I have doubts about whether this feature will be implemented in vte ever, but one thing is for sure and it's that this patch takes the wrong approach.
Behdad: > it's that this patch takes the wrong approach. Can you elaborate what would be the right approach here (to get a patch that would be accepted)?
(In reply to comment #8) > Behdad: > > it's that this patch takes the wrong approach. > > Can you elaborate what would be the right approach here (to get a patch that > would be accepted)? Umm, tell me about it! I've been thinking about this problem literally for years! Basically, a solution will have the following properties: - Code-wise, does not modify the vte backends, but the vte internal buffer. RTL text should be considered so by the terminal logic. - Probably should not do anything in alternate screen mode. Someone should check how Apple's terminal handles this. The thing is, the application running in the terminal does not expect you to reorder text. Your favorite editor for example has its own idea of where on the screen each character shows up. You can't just change that and expect everything to work. One may want to checkout BiCon that I linked above. There's also something called ECMA TR53 that is supposedly a spec for bidi text in terminals. No one knows if it was ever implemented.
>I've been thinking about this problem literally for years! thanks Behdad for that effort, I know that you used to contribute to Arabeyes.org > Apple's terminal Apple's terminal is not open source but KDE 4 terminal works perfectly with Arabic so it's better to consider it before considering Apple see this image http://www.linuxac.org/forum/attachment.php?attachmentid=2630&stc=1&d=1206636268 > - Code-wise, does not modify the vte backends, but the vte internal buffer. RTL text should be considered so by the terminal logic. could you please point me to the .c files or show me some documentation or any hint because when I did the patch a did a grep on the source tree to know where the text is rendered and fix that (in a way that you consider to be wrong) > One may want to checkout BiCon that I linked above. BiCon is dead, it has several problems 1. it only works with its children process, it should be made a daemon 2. it seems that it has memory problems leakage or something, and I can't trust an alpha software like it to be the parent process of RHGB or whatever I used to like acon in mandriva but that one does not support UTF-8
(In reply to comment #10) > > Apple's terminal > Apple's terminal is not open source I just want to know how they handle it functionally. > but KDE 4 terminal works perfectly with Arabic > so it's better to consider it before considering Apple > see this image > http://www.linuxac.org/forum/attachment.php?attachmentid=2630&stc=1&d=1206636268 Sure. I tried it, and it has all the problems that is hard to solve. Try selecting the RTL text, or move cursor around it. Or do RTL in vim or emacs.. Those are the really hard parts and I'm yet to see a terminal that gets them right. I *think* apple at least gets selection right. > > One may want to checkout BiCon that I linked above. > > BiCon is dead, it has several problems > 1. it only works with its children process, it should be made a daemon > 2. it seems that it has memory problems leakage or something, and I can't trust > an alpha software like it to be the parent process of RHGB or whatever > > I used to like acon in mandriva but that one does not support UTF-8 Well, BiCon is five times more reliable code than acon ever was...
I don't mind having one more open bug...
Hello, Any plans to implementation arabic in VTE correctly for Gnome 3 ?
(In reply to comment #13) > Hello, > > Any plans to implementation arabic in VTE correctly for Gnome 3 ? No.
Hi, Since the VTE does not support RLT languages, wouldn't it be possible to really do so and say 'I do not support this locale, falling back to standard locale "C"' ? I agree, it is not a fix at all, but at least users would have readable english instead of unreadable RTL language. The counterpart would be that all application launched from the VTE will inherit from this locale. what's your opinion ?
@JB Lallement That will be great if there is no other solution. @Behdad I've heard that you are close to finish Harf-Buzz and it will fix this problem. should we wait or try what @JB Lallement suggested? please take a look at: https://bugs.launchpad.net/ubuntu/+source/vte/+bug/263822
HarfBuzz doesn't help much with this bug, no. Regarding JB's suggestion, I'm not sure what you are suggesting. Changing LANG inside the terminal is not an option, no.
*** Bug 512678 has been marked as a duplicate of this bug. ***
Can we implement Konsole's approach to this issue?
(In reply to comment #19) > Can we implement Konsole's approach to this issue? Would you mind sharing "Konsole's approach"?
Apparently this is an implementation of fribidi: http://lists.arabeyes.org/archives/developer/2002/May/msg00017.html There's also MLTerm that presents great support for both Asian and RTL languages. Kind regards, Yaron Shahrabani.
And the relevant code from Konsole can be found here: https://projects.kde.org/projects/kde/kde-baseapps/konsole/repository/revisions/master/entry/src/TerminalDisplay.cpp#L726
Still investigating, Apparently Qt is handling the text (using a special bidi mechanism) and Konsole does nothing special to make sure it looks good... Maybe the implementation should go deeper beyond VTE in order for this to work correctly... Kind regards, Yaron Shahrabani.
2013 without fix !!!!!!!!!!
(In reply to comment #24) > 2013 without fix !!!!!!!!!! Old KDE link not working any longer, please use this one: https://projects.kde.org/projects/kde/applications/konsole/repository/revisions/master/entry/src/TerminalDisplay.cpp#L728 (Konsole has it's own module now and it's not part of KDE-base)
*** Bug 579676 has been marked as a duplicate of this bug. ***
*** Bug 704657 has been marked as a duplicate of this bug. ***
(In reply to comment #2) > This WONTFIX in vte. Not in my sight. You can run BiCon inside vte to get a > similar behavior: > > > > http://www.arabeyes.org/project.php?proj=BiCon BiCon isn't solution , It's primitive.
(In reply to comment #9) > I've been thinking about this problem literally for years! > > Basically, a solution will have the following properties: > > - Code-wise, does not modify the vte backends, but the vte internal buffer. > RTL text should be considered so by the terminal logic. I don't understand what you mean by this :) I've been also thinking about it for a while. I have two ideas on my own: 1. Trying to get a behavior as close to a multiline text editing widget or a graphical bidi-aware editor as possible. A letter or escape sequence being printed by the console app would correspond to pressing the letter or the arrow key in the editor. It's unclear to me if left/right should be swapped and how backspace, delete characters, insert character etc. should work. The problem with this is that fullscreen apps (not knowing anything about RTL, and possibly heavily optimizing for updating as little as possible, e.g. jumping with the cursor and only updating a part of a word) have their own expectation of where the cursor is. Chances are that no matter how the details of this approach are worked out, there's always a sequence of operations where the actual displayed text goes totally of out sync with what the application would expect. E.g. just one label is RTL in an ncurses app, and pieces of text outside that label get corrupted as the label is updated. I'm not sure, it's a guts feeling that this approach just can't be worked out. 2. BiDi-ing for display only. Imagine kind of a double buffering where current VTE works as it works now except that its canvas is invisible, and a BiDi layer in front of this looks at the content and applies the BiDi algorithm for each paragraph and re-runs this every time the contents change. This has the obvious advantage that the screen won't ever get corrupted with any ncurses or similar apps, ncurses will only see the non-BiDi layer (moving the cursor, querying the cursor position etc. would all work on that one). Drawbacks include strange jumps of the visual cursor (e.g. in a text editor an LTR line followed by an RTL, moving the cursor vertically also moves it horizontally). Horizontal cursor movement would also cause jumps and have the Left arrow key = move backwards logically, Right arrow key = move forwards logically semantics instead of visual left/right, this is probably not as good as it could be, but is a reasonable compromise (is it?). vterowdata needs to be extended to keep track of LRM/LRE/LRO and friends. What happens currently to these characters? I guess they're zero width and hence are handled equally to combining characters, that is, kept along with the base char of the preceding cell. Would this be good enough? What's the preferred behavior if such a cell is overwritten? Is the current limit on combining chars large enough? What if such a character is output in the 1st column, without a base character? Currently it is dropped which is definitely not good, this needs to be special cased in vterowdata. Mouse click coordinates would have to be reverse-translated, both when highlighting text (for computing selection_start and selection_end with its current non-bidi semantics) and when forwarded to the app (obvious). > - Probably should not do anything in alternate screen mode. I don't think it's a good idea to do something differently in alternate screen mode. It's error prone, hard to maintain, hard to test. The "joe" text editor lets you configure whether you want to switch to alternate screen or use the normal one, I wouldn't be surprised if it was configurable in vim/emacs too. One thing we can do differently in alternate screen is to run the BiDi-algorithm for the physical lines instead of paragraphs. This might fix cases if a fullscreen app optimizes on autowrapping although it prints something irrelevant in the two lines. For the normal screen, we definitely need to run BiDi on the paragraphs so that wrapping lines look okay (and also for rewrap to work). > Someone should check how Apple's terminal handles this. Indeed, we need to check what other BiDi-aware terminals (and apps, if any) do. > One may want to checkout BiCon that I linked above. We've heard complaints that BiCon is buggy, and has been unmaintained for 10 years. But improving BiCon, reimplementing from scratch, or adding a BiDi layer to screen/tmux is worth considering. > There's also something > called ECMA TR53 that is supposedly a spec for bidi text in terminals. No one > knows if it was ever implemented. Let's not forget that this standard is 22 years old. It talks about escape sequences that influence the behavior of the terminal. Assuming no apps ever emit those escapes (to be verified) we should check whether LRM and friends can reasonably mapped to these (probably not) or if we're safe to drop parts of the spec that rely on any of these escapes ever being emitted, and see what the result is. After a quick glimpse I can see implicit (the default: the app doesn't do anything special for RTL) and explicit modes, as well as data component vs presentation component, I'd be happy if it actually turned out the same as my nr. 2 proposal :)
Hi, If anyone will get to implement bidi then please consider to force the slash character "/" as a RTL encoded character. I've tried BiCon and mlterm, both failing in this (mlterm is better in regards to bidi support). A better description of this issue can be found in mlterm's mailing list at http://sourceforge.net/p/mlterm/mailman/message/32504201/
(In reply to comment #30) > Hi, > > If anyone will get to implement bidi then please consider to force the slash > character "/" as a RTL encoded character. > I've tried BiCon and mlterm, both failing in this (mlterm is better in regards > to bidi support). > > A better description of this issue can be found in mlterm's mailing list at > http://sourceforge.net/p/mlterm/mailman/message/32504201/ Sorry for dual posting, I meant "/" as a LTR encoded character
Created attachment 279300 [details] Driving Output into FriBIDI Both Mlterm and Bicon useless ... Even Konsole has a secondary solve. In my theory we must add Bidi (friBIDI) support to Linux itself .. ttyUnits must drive output to fribidi before out them .. To try this solve you can use this cmmand (see screenshot) : echo المنزل | fribidi
(In reply to comment #32) > Created an attachment (id=279300) [details] > Driving Output into FriBIDI > > Both Mlterm and Bicon useless ... Even Konsole has a secondary solve. > > In my theory we must add Bidi (friBIDI) support to Linux itself .. ttyUnits > must drive output to fribidi before out them .. > > To try this solve you can use this cmmand (see screenshot) : > > echo المنزل | fribidi That's what BiCon does, you know? ;) I know because I wrote both (major parts of) FriBidi and BiCon.
(In reply to comment #30) > Hi, > > If anyone will get to implement bidi then please consider to force the slash > character "/" as a RTL encoded character. We can't just decide to change a character's bidi property. That is specified by Unicode and best if everyone respects that. Otherwise same text will render differently in different environments. > I've tried BiCon and mlterm, both failing in this (mlterm is better in regards > to bidi support). > > A better description of this issue can be found in mlterm's mailing list at > http://sourceforge.net/p/mlterm/mailman/message/32504201/
(In reply to comment #33) > (In reply to comment #32) > > Created an attachment (id=279300) [details] [details] > > Driving Output into FriBIDI > > > > Both Mlterm and Bicon useless ... Even Konsole has a secondary solve. > > > > In my theory we must add Bidi (friBIDI) support to Linux itself .. ttyUnits > > must drive output to fribidi before out them .. > > > > To try this solve you can use this cmmand (see screenshot) : > > > > echo المنزل | fribidi > > That's what BiCon does, you know? ;) I know because I wrote both (major parts > of) FriBidi and BiCon. How about make it Auto in tty or in Bash ??
(In reply to comment #35) > (In reply to comment #33) > > (In reply to comment #32) > > > Created an attachment (id=279300) [details] [details] [details] > > > Driving Output into FriBIDI > > > > > > Both Mlterm and Bicon useless ... Even Konsole has a secondary solve. > > > > > > In my theory we must add Bidi (friBIDI) support to Linux itself .. ttyUnits > > > must drive output to fribidi before out them .. > > > > > > To try this solve you can use this cmmand (see screenshot) : > > > > > > echo المنزل | fribidi > > > > That's what BiCon does, you know? ;) I know because I wrote both (major parts > > of) FriBidi and BiCon. > > How about make it Auto in tty or in Bash ?? You can do that using whatever's launching your shell, to make it launch bicon instead.
Bicon doesn't solve this problem.
FWIW I've started using bicon inside my terminal. Please grab latest bicon from my github: https://github.com/behdad/bicon Build it, then in gnome-terminal's preferences, set "Run a custom command instead of my shell" and point it to the binary built inside the bicon tree. Eg: /home/behdad/src/github/bicon/bicon/bicon.bin Let me know how it goes. I fixed bicon to fix window sizing issues.
I tested 0.4.0 and it works fine. Could you attach the tarball on github? https://help.github.com/articles/creating-releases https://github.com/ibus/ibus-anthy/releases The default taraball is not made by 'make dist' and the name is %{version}.tar.gz but not %{name}-%{version}.tar.gz.
Done: https://github.com/behdad/bicon/releases/tag/0.4
Thanks.
*** Bug 759008 has been marked as a duplicate of this bug. ***
*** Bug 781952 has been marked as a duplicate of this bug. ***
BiDi / RTL support is being worked on at https://gitlab.gnome.org/GNOME/vte/issues/53. I'm closing this bug in favor of the new one.