Hi,

Thanks for this feature request!

I have mixed feelings about it.

First of all, I believe that usually it should be the app doing the coloring. Even if the app doesn't do that on its own, piping the output through a colorizer might be a better approach (although it has some disadvantages too). On the other hand, I can also see cases where such a feature could really be useful. E.g. I used to group digits by 3 in iTerm2 using the equivalent of this feature there, see https://gitlab.com/gnachman/iterm2/issues/1814.

There's absolutely no way a hardcoded set of rules could be sufficient to most users. While gnome-terminal could ship a default set (or a few sets) of rules, the default should be no highlighting, plus there should be a convenient way for users to edit the rules, add their own ones, or even share their rules (export/import).

gnome-terminal doesn't seem to be the kind of terminal emulator offering tons of features, it tends to offer fewer and keep its settings quite simple. Compared to how its (Profile) Preferences dialogs look like, this would be a quite complex addition. Another possibility to consider is to add the feature to VTE only, and let other VTE-based apps (e.g. Tilix) finish the work if they wish to.

As I mentioned in the referred iTerm2 bugreport, the feature should go to the displaying part of VTE, and definitely not where the input stream is processed.

I would think of an approach similar to those mentioned in the color mangling API bug 758368, probably even passing other attributes (bold etc.) to the callback to freely alter. There could be a separate callback for each regex registered.

We do have regex matching implemented for URLs, but that only looks at the mouse position. We'd need to change this or add another one that always keeps an eye on the entire screen, plus a few lines of context (bug 754936). Any change in the text of a given paragraph should trigger re-running the regex matches for the entire paragraph, and potentially cause all its cells to be invalidated (repainted).

I guess it would be nice to have it, but requires a considerable amount of work, whereas great colorizer tools (where you might choose the proper one depending on the particular utility you're about to colorize) are already available.

I'm curious to hear gnome-terminal's main developer's thoughts :-)

PS. Why isn't the first "checking" highlighted in the screenshot?

IMHO this is job for a processing filter running inside the terminal, not for vte.

Relevant discussion is happening in https://gitlab.gnome.org/GNOME/vte/issues/117.

> [chpe] IMHO this is job for a processing filter running inside the terminal, not for vte.

I'm not that certain.

Allegedly there are some great colorizers out there. However, they're probably problematic in some ways, e.g. might have to delay the output until they can tell for sure whether a word is to be highlighted or not (so probably they do line buffering), they should pretend towards the utility that they're colorizing that it's a terminal so that they flush after every line of the output, etc.

And if these are all figured out, it still only works for utilities that simply produce some output. It's likely to break if there's some escape sequence (e.g. color change) in between, and it almost certainly breaks if there's cursor positioning involved. In particular, it's absolutely hopeless to have such automated syntax highlighting over a fullscreen app (e.g. ncurses, vim/emacs etc.).

Syntax highlighting done by the terminal has the advantage that it's the final string that matters regardless of cursor motion and partially overrides and whatnot that happened in between, and that the rules can be altered or switched on/off retroactively.

(Coincidentally someone asked this just today at https://unix.stackexchange.com/q/515553/53656.)

> [me] requires a considerable amount of work

With the forthcoming introduction of RingView (for BiDi), some of this work will be done.

So, while it's certainly not the most important feature for a terminal emulator, it's something one can easily live without, and has (limited) substitute solutions, I think it's still a nice feature to have. And technically doing syntax highlighting is pretty similar to highlighting all the search matches at once (another nice feature); if we do any of these two then we can just as easily do both at once.

> And if these are all figured out, it still only works for utilities that simply 
> produce some output. It's likely to break if there's some escape sequence (e.g. 
> color change) in between, and it almost certainly breaks if there's cursor 
> positioning involved. In particular, it's absolutely hopeless to have such 
> automated syntax highlighting over a fullscreen app (e.g. ncurses, vim/emacs \
> etc.).

I'd say that this only makes sense for applications that output plain text, not ones that use escape sequences. Anything involving cursor movement, attribute changes, and certainly ncurses apps, shouldn't get this auto-highlighting.

> And technically doing syntax highlighting is pretty similar to highlighting all 
> the search matches at once (another nice feature); if we do any of these two 
> then we can just as easily do both at once.

As long as you want to highlight only in one colour, yes. 'error'->red, 'warning'->yellow already doesn't fit into that model. But if a simple highlighter comes out with only one colour, then integrating that with search-highlight-all (e.g. by having a built-in regex that does syntax highlighting) then that's ok with me.

(In reply to Christian Persch from comment #4)

> I'd say that this only makes sense for applications that output plain text,
> not ones that use escape sequences. Anything involving cursor movement,
> attribute changes, and certainly ncurses apps, shouldn't get this
> auto-highlighting.

Not sure if it only makes sense in that case. See e.g. the number grouping idea I linked from the top of comment 3 which would (or in case of iTerm2: does) work with ncurses apps as well.

Not highlighting across attribute changes, cursor movements etc. also leads to a less consistent user experience, with apparent bugs like "I asked gnome-terminal to highlight all the "error" words and it highlights most of them but not all, why?". Due to how ncurses optimizes the hell out of screen updates, in an interactive program some of the "error" words will be printed in a single run (hence highlighted by g-t) while some will be printed by let's say overwriting "elephant"'s last seven letters by "rror in" or so, and thus not highlighted – an absolutely inconsistent (as the user perceives: buggy) behavior.


There are two fundamentally different approaches to syntax highlighting:

(1) match against the input stream as it comes in;
(2) match against the onscreen contents whenever it gets painted.


Any external colorizer tool would likely go with (1), since performing actual terminal emulation would be way too heavy work with them.

*If* we are to implement one of these two in VTE (which is still arguable if we want or not), I find approach (2) to be absolutely superior for the following reasons:

- I believe it's significantly easier to implement (especially with forthcoming RingView that we'll need anyway for BiDi).

- Provides a consistent, bugfree user experience (no missing matches due to ncurses etc. which is hard to explain and reason for).

- More flexible, as it allows arbitrary retroactive changes to the rules.

- Cheaper, as only a subset of the data needs to be examined.


[A completely different feature that would require approach (1) would be the kind of one-time triggers like "if you encounter the word «error», beep". This is a different story since it's not about the visual representation, e.g. you can't just scan the piece of text that makes it into display pixels, you need to scan the entire input stream and you need to take action whenever the received input matches a certain pattern. This is an entirely different story, and I'm personally not that much interested in this one. And this one is pretty easily doable using an external utility, without compromises such as delayed outputting.]


> > And technically doing syntax highlighting is pretty similar to highlighting all 
> > the search matches at once (another nice feature); if we do any of these two 
> > then we can just as easily do both at once.
> 
> As long as you want to highlight only in one colour, yes. 'error'->red,
> 'warning'->yellow already doesn't fit into that model.

I'm not sure why it doesn't fit.

I mean, obviously if you implement "highlight all" with one particular color then you don't get "error->red, warning->yellow" style coloring literally immediately out of the box, but you get it (using approach (2)) with very little work.


My point essentially is:

- If we do syntax highlighting, it needs to be approach (2).

- "Highlight all search matches" and "syntax highlighting approach (2)" are so very close to each other that if we address either of them, we should probably address both.

Another great use case occurred to me for syntax highlighting.

I just spent a few minutes pondering why this fragment in a shell script:
  VTE_DEBUG="${VTE_DEBUG-bidi,ringview}"
didn't preserve the empty value when I invoked it as
  VTE_DEUBG= myscript.sh

The answer is a typo, and it's by far not the first time I make this very same typo. I think many people have a couple of words that they often misspell in a particular way.

I would so totally add a rule that paints the word "deubg" in red and even blinks it for me. This would then work both in my terminal-based text editor and at the shell prompt.

(Note that the forthcoming ringview infrastructure could easily accomodate proper spell checking too, but probably that's not a good idea in a context where there are way too many technical words, variables names etc., and the content is often dropped.)

And yet another great use case:

Highlight NBSP (see e.g. https://askubuntu.com/q/1147026/398785).

Created attachment 374221 [details] [review]
Proof of concept

Here's a proof of concept patch.

Goes on top of current git master + vtebidi-v1 from https://gitlab.gnome.org/GNOME/vte/issues/53 (because this feature relies on a piece of infrastructure that's introduced by the forthcoming BiDi support).

The rules are hardwired in the patch, but should be pretty straightforward to modify. Only case insensitive(*) string searching is performed at the moment, no regex matching, but this shouldn't be hard to extend either.

(*) Case insensitivitiy only works for 7-bit ASCII characters.

Other than these inconveniences, it's believed to be rock solid and bugfree. :)

Created attachment 374225 [details] [review]
Proof of concept, v2

Just a trivial update to resolve a conflict.

Goes on top of current git master (17cf94a83), which now includes the necessary RingView infrastructure.

(In reply to Egmont Koblinger from comment #6)

> and the content is often dropped.

Typo: I meant "cropped" (i.e. partially offscreen).

> attachment 374225 [details] [review]

This should use a regex for highlighting. 

For changing the attributes, the regex could use the (*MARK:name) syntax to record which branch matched, and then either parse the mark name as attribute directly, or (better) make name a simple number, and use that as an index into a passed-in array of attributes to apply.

-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/gnome-terminal/-/issues/7771.