After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 321490 - (vtebidi) arabic, hebrew: character alignment not working properly
(vtebidi)
arabic, hebrew: character alignment not working properly
Status: RESOLVED OBSOLETE
Product: vte
Classification: Core
Component: general
unspecified
Other opensolaris
: Normal normal
: ---
Assigned To: VTE Maintainers
VTE Maintainers
[wontfix]
: 512678 579676 704657 759008 781952 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2005-11-15 02:44 UTC by Takao Fujiwara
Modified: 2019-01-29 11:17 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Snapshot for gnome-terminal (23.20 KB, image/png)
2005-11-15 02:45 UTC, Takao Fujiwara
  Details
pacth to vte-0.16.13 which enables BiDi and shaping (8.88 KB, patch)
2008-05-18 21:47 UTC, muayyad alsadi
none Details | Review
fix bidi problems (apply with patch -p1) (5.27 KB, patch)
2008-07-15 13:25 UTC, muayyad alsadi
rejected Details | Review
Driving Output into FriBIDI (529.65 KB, image/png)
2014-06-26 09:17 UTC, Mosaab Alzoubi
  Details

Description Takao Fujiwara 2005-11-15 02:44:13 UTC
Arabic writing direction is not correct.
Actual behavior: from left to right.
Expected behavior: from right to left.
Comment 1 Takao Fujiwara 2005-11-15 02:45:45 UTC
Created attachment 54765 [details]
Snapshot for gnome-terminal

The snapshot shows I input 'a', 'b', 'c', 'd', 'e'.
Comment 2 Behdad Esfahbod 2006-04-28 02:33:36 UTC
This WONTFIX in vte.  Not in my sight.  You can run BiCon inside vte to get a similar behavior:

  

  http://www.arabeyes.org/project.php?proj=BiCon
Comment 3 muayyad alsadi 2008-05-18 21:47:45 UTC
Created attachment 111119 [details] [review]
pacth to vte-0.16.13 which enables BiDi and shaping

this patch will disable xft2 because I added the support to pango and pangox backends only
Comment 4 muayyad alsadi 2008-05-18 21:51:51 UTC
Salam,
I fix and it works

apply this patch to version vte-0.16.13
cd vte-0.16.13
patch -p1 < vte-0.16.13-2.patch

and then rebuild it and it will work
gnome-terminal and rhgb will work as a result

my patch does the following
1. disable xft2 (till similar procedure is done to it)
2. make pango backend handles request line by line if possible not char by char

yes this is a primitive support like the one in konsole of kde 3.x
there are some known problem like when you are typing because only one char gets updated


the problem is due to that fact vte does not update every thing
it only updates the changes
but if there is a trick to add a timeout the update the entire screen
every second
it will be good (I don't know how, because I don't know the logic of vte)

Comment 5 muayyad alsadi 2008-05-18 21:54:07 UTC
Salam,
I fix and it works

apply this patch to version vte-0.16.13
cd vte-0.16.13
patch -p1 < vte-0.16.13-2.patch

and then rebuild it and it will work
gnome-terminal and rhgb will work as a result

my patch does the following
1. disable xft2 (till similar procedure is done to it)
2. make pango backend handles request line by line if possible not char by char

yes this is a primitive support like the one in konsole of kde 3.x
there are some known problem like when you are typing because only one char gets updated


the problem is due to that fact vte does not update every thing
it only updates the changes
but if there is a trick to add a timeout the update the entire screen
every second
it will be good (I don't know how, because I don't know the logic of vte)

Comment 6 muayyad alsadi 2008-07-15 13:25:27 UTC
Created attachment 114595 [details] [review]
fix bidi problems (apply with patch -p1)

better than the previous patch
Comment 7 Behdad Esfahbod 2008-07-15 14:26:52 UTC
I have doubts about whether this feature will be implemented in vte ever, but one thing is for sure and it's that this patch takes the wrong approach.
Comment 8 André Klapper 2008-09-04 20:50:20 UTC
Behdad: 
> it's that this patch takes the wrong approach.

Can you elaborate what would be the right approach here (to get a patch that would be accepted)?
Comment 9 Behdad Esfahbod 2008-09-05 22:00:00 UTC
(In reply to comment #8)
> Behdad: 
> > it's that this patch takes the wrong approach.
> 
> Can you elaborate what would be the right approach here (to get a patch that
> would be accepted)?

Umm, tell me about it!  I've been thinking about this problem literally for years!

Basically, a solution will have the following properties:

  - Code-wise, does not modify the vte backends, but the vte internal buffer.  RTL text should be considered so by the terminal logic.

  - Probably should not do anything in alternate screen mode.  Someone should check how Apple's terminal handles this.  The thing is, the application running in the terminal does not expect you to reorder text.  Your favorite editor for example has its own idea of where on the screen each character shows up.  You can't just change that and expect everything to work.

One may want to checkout BiCon that I linked above.  There's also something called ECMA TR53 that is supposedly a spec for bidi text in terminals.  No one knows if it was ever implemented.
Comment 10 muayyad alsadi 2008-09-06 14:49:08 UTC
>I've been thinking about this problem literally for
years!

thanks Behdad for that effort, I know that you used to contribute to Arabeyes.org

> Apple's terminal
Apple's terminal is not open source
but KDE 4 terminal works perfectly with Arabic
so it's better to consider it before considering Apple
see this image
http://www.linuxac.org/forum/attachment.php?attachmentid=2630&stc=1&d=1206636268

>  - Code-wise, does not modify the vte backends, but the vte internal buffer. 
RTL text should be considered so by the terminal logic.
could you please point me to the .c files
or show me some documentation or any hint

because when I did the patch a did a grep on the source tree to know where the text is rendered and fix that (in a way that you consider to be wrong)

> One may want to checkout BiCon that I linked above.

BiCon is dead, it has several problems
1. it only works with its children process, it should be made a daemon
2. it seems that it has memory problems leakage or something, and I can't trust an alpha software like it to be the parent process of RHGB or whatever

I used to like acon in mandriva but that one does not support UTF-8
Comment 11 Behdad Esfahbod 2008-09-08 04:10:19 UTC
(In reply to comment #10)

> > Apple's terminal
> Apple's terminal is not open source

I just want to know how they handle it functionally.

> but KDE 4 terminal works perfectly with Arabic
> so it's better to consider it before considering Apple
> see this image
> http://www.linuxac.org/forum/attachment.php?attachmentid=2630&stc=1&d=1206636268

Sure.  I tried it, and it has all the problems that is hard to solve.  Try selecting the RTL text, or move cursor around it.  Or do RTL in vim or emacs.. Those are the really hard parts and I'm yet to see a terminal that gets them right.  I *think* apple at least gets selection right.


> > One may want to checkout BiCon that I linked above.
> 
> BiCon is dead, it has several problems
> 1. it only works with its children process, it should be made a daemon
> 2. it seems that it has memory problems leakage or something, and I can't trust
> an alpha software like it to be the parent process of RHGB or whatever
> 
> I used to like acon in mandriva but that one does not support UTF-8

Well, BiCon is five times more reliable code than acon ever was...
Comment 12 Behdad Esfahbod 2008-09-08 04:10:42 UTC
I don't mind having one more open bug...
Comment 13 أحمد المحمودي (Ahmed El-Mahmoudy) 2009-07-19 19:35:58 UTC
Hello,

  Any plans to implementation arabic in VTE correctly for Gnome 3 ?
Comment 14 Behdad Esfahbod 2009-07-19 19:38:22 UTC
(In reply to comment #13)
> Hello,
> 
>   Any plans to implementation arabic in VTE correctly for Gnome 3 ?

No.
Comment 15 JB Lallement 2010-03-08 17:25:11 UTC
Hi,

Since the VTE does not support RLT languages, wouldn't it be possible to really do so and say 'I do not support this locale, falling back to standard locale "C"' ? 
I agree, it is not a fix at all, but at least users would have readable english instead of unreadable RTL language.

The counterpart would be that all application launched from the VTE will inherit from this locale.

what's your opinion ?
Comment 16 Usama Akkad 2010-04-02 19:37:43 UTC
@JB Lallement
That will be great if there is no other solution.

@Behdad
I've heard that you are close to finish Harf-Buzz and it will fix this problem. should we wait or try what @JB Lallement suggested?

please take a look at:
https://bugs.launchpad.net/ubuntu/+source/vte/+bug/263822
Comment 17 Behdad Esfahbod 2010-04-06 03:51:46 UTC
HarfBuzz doesn't help much with this bug, no.

Regarding JB's suggestion, I'm not sure what you are suggesting.  Changing LANG inside the terminal is not an option, no.
Comment 18 Behdad Esfahbod 2010-12-02 01:17:13 UTC
*** Bug 512678 has been marked as a duplicate of this bug. ***
Comment 19 Yaron Shahrabani 2012-11-06 11:10:33 UTC
Can we implement Konsole's approach to this issue?
Comment 20 André Klapper 2012-11-06 12:02:44 UTC
(In reply to comment #19)
> Can we implement Konsole's approach to this issue?

Would you mind sharing "Konsole's approach"?
Comment 21 Yaron Shahrabani 2012-11-06 12:27:12 UTC
Apparently this is an implementation of fribidi: http://lists.arabeyes.org/archives/developer/2002/May/msg00017.html

There's also MLTerm that presents great support for both Asian and RTL languages.

Kind regards,
Yaron Shahrabani.
Comment 22 Yaron Shahrabani 2012-11-06 12:52:18 UTC
And the relevant code from Konsole can be found here:
https://projects.kde.org/projects/kde/kde-baseapps/konsole/repository/revisions/master/entry/src/TerminalDisplay.cpp#L726
Comment 23 Yaron Shahrabani 2012-11-07 14:20:08 UTC
Still investigating,
Apparently Qt is handling the text (using a special bidi mechanism) and Konsole does nothing special to make sure it looks good...

Maybe the implementation should go deeper beyond VTE in order for this to work correctly...

Kind regards,
Yaron Shahrabani.
Comment 24 Mosaab Alzoubi 2013-08-01 02:25:52 UTC
2013 without fix !!!!!!!!!!
Comment 25 Yaron Shahrabani 2013-08-04 07:46:47 UTC
(In reply to comment #24)
> 2013 without fix !!!!!!!!!!

Old KDE link not working any longer, please use this one:
https://projects.kde.org/projects/kde/applications/konsole/repository/revisions/master/entry/src/TerminalDisplay.cpp#L728

(Konsole has it's own module now and it's not part of KDE-base)
Comment 26 Christian Persch 2013-09-14 12:59:03 UTC
*** Bug 579676 has been marked as a duplicate of this bug. ***
Comment 27 Christian Persch 2013-09-14 12:59:22 UTC
*** Bug 704657 has been marked as a duplicate of this bug. ***
Comment 28 Mosaab Alzoubi 2013-11-09 11:58:13 UTC
(In reply to comment #2)
> This WONTFIX in vte.  Not in my sight.  You can run BiCon inside vte to get a
> similar behavior:
> 
> 
> 
>   http://www.arabeyes.org/project.php?proj=BiCon

BiCon isn't solution , It's primitive.
Comment 29 Egmont Koblinger 2014-06-10 01:37:02 UTC
(In reply to comment #9)

> I've been thinking about this problem literally for years!
> 
> Basically, a solution will have the following properties:
> 
>   - Code-wise, does not modify the vte backends, but the vte internal buffer. 
> RTL text should be considered so by the terminal logic.

I don't understand what you mean by this :)


I've been also thinking about it for a while.  I have two ideas on my own:

1. Trying to get a behavior as close to a multiline text editing widget or a graphical bidi-aware editor as possible.  A letter or escape sequence being printed by the console app would correspond to pressing the letter or the arrow key in the editor.  It's unclear to me if left/right should be swapped and how backspace, delete characters, insert character etc. should work.

The problem with this is that fullscreen apps (not knowing anything about RTL, and possibly heavily optimizing for updating as little as possible, e.g. jumping with the cursor and only updating a part of a word) have their own expectation of where the cursor is.  Chances are that no matter how the details of this approach are worked out, there's always a sequence of operations where the actual displayed text goes totally of out sync with what the application would expect.  E.g. just one label is RTL in an ncurses app, and pieces of text outside that label get corrupted as the label is updated.  I'm not sure, it's a guts feeling that this approach just can't be worked out.

2.  BiDi-ing for display only.  Imagine kind of a double buffering where current VTE works as it works now except that its canvas is invisible, and a BiDi layer in front of this looks at the content and applies the BiDi algorithm for each paragraph and re-runs this every time the contents change.  This has the obvious advantage that the screen won't ever get corrupted with any ncurses or similar apps, ncurses will only see the non-BiDi layer (moving the cursor, querying the cursor position etc. would all work on that one).  Drawbacks include strange jumps of the visual cursor (e.g. in a text editor an LTR line followed by an RTL, moving the cursor vertically also moves it horizontally).  Horizontal cursor movement would also cause jumps and have the Left arrow key = move backwards logically, Right arrow key = move forwards logically semantics instead of visual left/right, this is probably not as good as it could be, but is a reasonable compromise (is it?).

vterowdata needs to be extended to keep track of LRM/LRE/LRO and friends.  What happens currently to these characters?  I guess they're zero width and hence are handled equally to combining characters, that is, kept along with the base char of the preceding cell.  Would this be good enough?  What's the preferred behavior if such a cell is overwritten?  Is the current limit on combining chars large enough?  What if such a character is output in the 1st column, without a base character?  Currently it is dropped which is definitely not good, this needs to be special cased in vterowdata.

Mouse click coordinates would have to be reverse-translated, both when highlighting text (for computing selection_start and selection_end with its current non-bidi semantics) and when forwarded to the app (obvious).

>   - Probably should not do anything in alternate screen mode.

I don't think it's a good idea to do something differently in alternate screen mode.  It's error prone, hard to maintain, hard to test.  The "joe" text editor lets you configure whether you want to switch to alternate screen or use the normal one, I wouldn't be surprised if it was configurable in vim/emacs too.

One thing we can do differently in alternate screen is to run the BiDi-algorithm for the physical lines instead of paragraphs.  This might fix cases if a fullscreen app optimizes on autowrapping although it prints something irrelevant in the two lines.  For the normal screen, we definitely need to run BiDi on the paragraphs so that wrapping lines look okay (and also for rewrap to work).

>  Someone should check how Apple's terminal handles this.

Indeed, we need to check what other BiDi-aware terminals (and apps, if any) do.

> One may want to checkout BiCon that I linked above.

We've heard complaints that BiCon is buggy, and has been unmaintained for 10 years.  But improving BiCon, reimplementing from scratch, or adding a BiDi layer to screen/tmux is worth considering.

> There's also something
> called ECMA TR53 that is supposedly a spec for bidi text in terminals.  No one
> knows if it was ever implemented.

Let's not forget that this standard is 22 years old.

It talks about escape sequences that influence the behavior of the terminal.  Assuming no apps ever emit those escapes (to be verified) we should check whether LRM and friends can reasonably mapped to these (probably not) or if we're safe to drop parts of the spec that rely on any of these escapes ever being emitted, and see what the result is.  After a quick glimpse I can see implicit (the default: the app doesn't do anything special for RTL) and explicit modes, as well as data component vs presentation component, I'd be happy if it actually turned out the same as my nr. 2 proposal :)
Comment 30 delete_my_account 2014-06-25 16:02:17 UTC
Hi,

If anyone will get to implement bidi then please consider to force the slash character "/" as a RTL encoded character.
I've tried BiCon and mlterm, both failing in this (mlterm is better in regards to bidi support).

A better description of this issue can be found in mlterm's mailing list at
http://sourceforge.net/p/mlterm/mailman/message/32504201/
Comment 31 delete_my_account 2014-06-25 16:04:22 UTC
(In reply to comment #30)
> Hi,
> 
> If anyone will get to implement bidi then please consider to force the slash
> character "/" as a RTL encoded character.
> I've tried BiCon and mlterm, both failing in this (mlterm is better in regards
> to bidi support).
> 
> A better description of this issue can be found in mlterm's mailing list at
> http://sourceforge.net/p/mlterm/mailman/message/32504201/

Sorry for dual posting, I meant "/" as a LTR encoded character
Comment 32 Mosaab Alzoubi 2014-06-26 09:17:30 UTC
Created attachment 279300 [details]
Driving Output into FriBIDI

Both Mlterm and Bicon useless ... Even Konsole has a secondary solve.

In my theory we must add Bidi (friBIDI) support to Linux itself .. ttyUnits must drive output to fribidi before out them ..

To try this solve you can use this cmmand (see screenshot) :

echo المنزل | fribidi
Comment 33 Behdad Esfahbod 2014-07-04 19:32:26 UTC
(In reply to comment #32)
> Created an attachment (id=279300) [details]
> Driving Output into FriBIDI
> 
> Both Mlterm and Bicon useless ... Even Konsole has a secondary solve.
> 
> In my theory we must add Bidi (friBIDI) support to Linux itself .. ttyUnits
> must drive output to fribidi before out them ..
> 
> To try this solve you can use this cmmand (see screenshot) :
> 
> echo المنزل | fribidi

That's what BiCon does, you know? ;)  I know because I wrote both (major parts of) FriBidi and BiCon.
Comment 34 Behdad Esfahbod 2014-07-04 19:33:25 UTC
(In reply to comment #30)
> Hi,
> 
> If anyone will get to implement bidi then please consider to force the slash
> character "/" as a RTL encoded character.

We can't just decide to change a character's bidi property.  That is specified by Unicode and best if everyone respects that.  Otherwise same text will render differently in different environments.


> I've tried BiCon and mlterm, both failing in this (mlterm is better in regards
> to bidi support).
> 
> A better description of this issue can be found in mlterm's mailing list at
> http://sourceforge.net/p/mlterm/mailman/message/32504201/
Comment 35 Mosaab Alzoubi 2014-07-04 23:03:35 UTC
(In reply to comment #33)
> (In reply to comment #32)
> > Created an attachment (id=279300) [details] [details]
> > Driving Output into FriBIDI
> > 
> > Both Mlterm and Bicon useless ... Even Konsole has a secondary solve.
> > 
> > In my theory we must add Bidi (friBIDI) support to Linux itself .. ttyUnits
> > must drive output to fribidi before out them ..
> > 
> > To try this solve you can use this cmmand (see screenshot) :
> > 
> > echo المنزل | fribidi
> 
> That's what BiCon does, you know? ;)  I know because I wrote both (major parts
> of) FriBidi and BiCon.

How about make it Auto in tty or in Bash ??
Comment 36 Behdad Esfahbod 2014-07-14 19:11:03 UTC
(In reply to comment #35)
> (In reply to comment #33)
> > (In reply to comment #32)
> > > Created an attachment (id=279300) [details] [details] [details]
> > > Driving Output into FriBIDI
> > > 
> > > Both Mlterm and Bicon useless ... Even Konsole has a secondary solve.
> > > 
> > > In my theory we must add Bidi (friBIDI) support to Linux itself .. ttyUnits
> > > must drive output to fribidi before out them ..
> > > 
> > > To try this solve you can use this cmmand (see screenshot) :
> > > 
> > > echo المنزل | fribidi
> > 
> > That's what BiCon does, you know? ;)  I know because I wrote both (major parts
> > of) FriBidi and BiCon.
> 
> How about make it Auto in tty or in Bash ??

You can do that using whatever's launching your shell, to make it launch bicon instead.
Comment 37 Mosaab Alzoubi 2014-07-14 23:15:13 UTC
Bicon doesn't solve this problem.
Comment 38 Behdad Esfahbod 2014-07-27 00:17:52 UTC
FWIW I've started using bicon inside my terminal.  Please grab latest bicon from my github:

  https://github.com/behdad/bicon

Build it, then in gnome-terminal's preferences, set "Run a custom command instead of my shell" and point it to the binary built inside the bicon tree.  Eg:

/home/behdad/src/github/bicon/bicon/bicon.bin

Let me know how it goes.  I fixed bicon to fix window sizing issues.
Comment 39 Takao Fujiwara 2014-07-28 05:03:45 UTC
I tested 0.4.0 and it works fine.
Could you attach the tarball on github?

https://help.github.com/articles/creating-releases
https://github.com/ibus/ibus-anthy/releases

The default taraball is not made by 'make dist' and the name is %{version}.tar.gz but not %{name}-%{version}.tar.gz.
Comment 40 Behdad Esfahbod 2014-07-28 20:21:41 UTC
Done: https://github.com/behdad/bicon/releases/tag/0.4
Comment 41 Takao Fujiwara 2014-07-29 03:11:18 UTC
Thanks.
Comment 42 Egmont Koblinger 2015-12-04 19:20:24 UTC
*** Bug 759008 has been marked as a duplicate of this bug. ***
Comment 43 Christian Persch 2017-04-29 21:40:42 UTC
*** Bug 781952 has been marked as a duplicate of this bug. ***
Comment 44 Egmont Koblinger 2019-01-29 11:17:11 UTC
BiDi / RTL support is being worked on at https://gitlab.gnome.org/GNOME/vte/issues/53.

I'm closing this bug in favor of the new one.