GNOME Bugzilla – Bug 612662
on screen keyboard support
Last modified: 2011-10-31 14:37:40 UTC
I suppose we should have some kind of built-in on-screen keyboard support. We'll want to support typing in the overview search box even when a physical keyboard isn't available. Will need to come up with some specific guidelines here I guess but my guess is that we'll want something that slides in at the bottom of the screen, sets struts, and compresses the content area (probably including the message tray). Thoughts?
http://live.gnome.org/GnomeShell/Design/Whiteboards/ScreenKeyboard
*** Bug 646656 has been marked as a duplicate of this bug. ***
Some thoughts for 3.2: Use cases: 1. Tablets and other large form-factor touch screens. (Not phones.) - To be easy to use, the buttons will have to be largish, making the keyboard as a whole take up a good chunk of the screen. We will probably want it to stay in place, at the bottom of the screen, as described above. - Does the workspace get smaller then, or just scroll/pan? If the keyboard is not always visible, it should be the latter. - For languages that would normally need an input method, simply providing an on-screen version of the existing physical keyboard is probably not the best solution. It's likely that to merge the "keyboard" and input method together in some way, which may end up looking like neither a keyboard nor the original input method. But not for 3.2. - The message tray needs to be above the keyboard, or else you can't type into chat bubbles. There should be an open/close tray button on the keyboard, since the hot corner/edge will be broken. 2. Accessibility (ie, Caribou replacement) - Pointer-based use is only one part of this. For full a11y it would eventually need to support switch-based use with various scan modes. We don't plan to implement that for 3.2, though others could. - Most (all?) existing a11y on-screen keyboards use a small floating keyboard, rather than a large fixed one; since you don't need to use your fingers it doesn't have to be as big, and you don't want to be zooming the pointer back and forth across the screen anyway. If the keyboard is going to float, it needs to avoid the caret. It is probably also good to stay *near* the caret though, so you can easily see the keyboard and the text at the same time. - The solution here for languages with complicated input methods may not be the same as the solution for the tablet use case. Misc notes: - Should have some sort of pluggable configuration. XML files or the like. We'll need community help to build these for various international layouts. - Probably don't want to reuse the xkb data; full keyboards are too bulky - It would be nice if entries could have keyboard mode hints (numeric, web address, email address, do/don't start capitalized, etc). This could perhaps be exposed via the input method interface, since that's where a lot of other information will be coming from. - https://fedorahosted.org/eekboard/ seems like a good starting point
(In reply to comment #3) > Some thoughts for 3.2: > > > Use cases: > > 1. Tablets and other large form-factor touch screens. (Not phones.) > > - To be easy to use, the buttons will have to be largish, > making the keyboard as a whole take up a good chunk of the > screen. We will probably want it to stay in place, at the > bottom of the screen, as described above. > > - Does the workspace get smaller then, or just scroll/pan? > If the keyboard is not always visible, it should be the > latter. After a brief talk with Joaquim, on Caribout you can define the size of each key on the defining file, so it would be easy to set a large size. Of course this doesn't solve the overview problem, themes and so on. > 2. Accessibility (ie, Caribou replacement) Replacement? Trying to get Caribou properly working on the shell is not an option? It was already discarded? (note that I'm not saying that the shell OSK should be Caribou, I'm talking about a11y aspects) > - Most (all?) existing a11y on-screen keyboards use a small > floating keyboard, rather than a large fixed one; since you > don't need to use your fingers it doesn't have to be as big, > and you don't want to be zooming the pointer back and forth > across the screen anyway. As I said, you can define that size. And for a11y requirements, it would be also good to define bigger keys if required. > Misc notes: > > - Should have some sort of pluggable configuration. XML files or the > like. We'll need community help to build these for various > international layouts. > > - Probably don't want to reuse the xkb data; full keyboards are > too bulky Caribou defines his keyboard using JSON/XML. (CCing Joaquim Rocha) > - It would be nice if entries could have keyboard mode hints (numeric, > web address, email address, do/don't start capitalized, etc). This > could perhaps be exposed via the input method interface, since > that's where a lot of other information will be coming from. Other interesting thing that could be exposed via the input method is text prediction, like this: http://www.joaquimrocha.com/2010/04/05/caribou-and-text-predictor-input-mode/ PS: also CCing Eitan, Caribou maintainer
Hi, I'm replying trusting that my changes to Caribou haven't been deeply changes as I haven't checked it lately. (In reply to comment #4) > > After a brief talk with Joaquim, on Caribout you can define the size of each > key on the defining file, so it would be easy to set a large size. Of course > this doesn't solve the overview problem, themes and so on. There is a predefined "standard" size for each key which AFAIR is hardcoded and you can set the width of each key. This width should be 1, 1.5, 3, etc. and will be multiplied by the standard size. We could make this standard size configurable. > > > 2. Accessibility (ie, Caribou replacement) > > Replacement? Trying to get Caribou properly working on the shell is not an > option? It was already discarded? (note that I'm not saying that the shell OSK > should be Caribou, I'm talking about a11y aspects) > > > - Most (all?) existing a11y on-screen keyboards use a small > > floating keyboard, rather than a large fixed one; since you > > don't need to use your fingers it doesn't have to be as big, > > and you don't want to be zooming the pointer back and forth > > across the screen anyway. > > As I said, you can define that size. And for a11y requirements, it would be > also good to define bigger keys if required. I think we can use Caribou for both purposes. We could define different profiles for Caribou: a11y and touchscreen for example. This would mean different keyboard layouts (already supported) and different behavior, standard size of keys. > > > Misc notes: > > > > - Should have some sort of pluggable configuration. XML files or the > > like. We'll need community help to build these for various > > international layouts. > > > > - Probably don't want to reuse the xkb data; full keyboards are > > too bulky > > Caribou defines his keyboard using JSON/XML. (CCing Joaquim Rocha) I think we should stay away from XKB layouts as they're intended for physical keyboards and we can be much more flexible in the screen. Besides, Caribou's layout system is very flexible. > > > - It would be nice if entries could have keyboard mode hints (numeric, > > web address, email address, do/don't start capitalized, etc). This > > could perhaps be exposed via the input method interface, since > > that's where a lot of other information will be coming from. > > Other interesting thing that could be exposed via the input method is text > prediction, like this: > > http://www.joaquimrocha.com/2010/04/05/caribou-and-text-predictor-input-mode/ > Yeah, I think that some things should be out of Caribou and one of them is text prediction like it is shown in that video. I still don't know much of these details of GNOME Shell but I think what would be cool was for Caribou to be treated as a special window in the way that it could appear anyway in the screen, independently from being inside a workspace or not. The behavior for the current (a11y) Caribou would be easy, leaving the workspace into the activities view would hide Caribou; if the search entry gets focused then it would appear near it. For the touchscreen Caribou some deeper tweaks would be necessary. It should likely occupy the whole width of the screen and about a half of its height and should overlap what's behind but make sure that the cursor wouldn't be hidden, which might require the widgets to be placed on the "visible" area above the screen. Let me know if I can help.
(In reply to comment #3) > Some thoughts for 3.2: > > > Use cases: > > 1. Tablets and other large form-factor touch screens. (Not phones.) > > - To be easy to use, the buttons will have to be largish, > making the keyboard as a whole take up a good chunk of the > screen. We will probably want it to stay in place, at the > bottom of the screen, as described above. > > - Does the workspace get smaller then, or just scroll/pan? > If the keyboard is not always visible, it should be the > latter. There's 2 possibilities here, one is the iPad style keyboard, which would work well for very (very very) focused applications (basically single-tasking). For this case, we'd have a keyboard that would show up "when needed" (eg. you focused a text entry). That would need to include zooming for things like web pages, possibly re-centering. The easier method is probably a drawer style keyboard, as used in other tablet OSes (like Windows 7, or tablet optimised MeeGo versions). The shell would have a way to drag the keyboard out. We might want animations for the resized windows. The bottom left-handside of the screen might provide a good hit-target for the keyboard, as it's currently empty, even when the message tray is shown. > - https://fedorahosted.org/eekboard/ seems like a good starting point That's the conclusion I came to as well.
(In reply to comment #4) > > 2. Accessibility (ie, Caribou replacement) > > Replacement? Trying to get Caribou properly working on the shell is not an > option? It was already discarded? (note that I'm not saying that the shell OSK > should be Caribou, I'm talking about a11y aspects) Hm... I'd thought that Caribou was not especially maintained at the moment, but looking at git, I guess that's not true. The big issue is that X windows can't be used in the overview, so the keyboard needs to be clutter-based and in-process (which in turn means it has to be either C or JavaScript because of how GObject bindings work). It is possible that this could be implemented via D-Bus like some of the status area icons, with the "business logic" happening out of process in caribou, and the UI happening in the shell, but I'm not sure that makes as much sense here as with the status icons. Also, I haven't had a chance to dive into the caribou source yet, so I don't have a good sense of everything it does.
(In reply to comment #7) > (In reply to comment #4) > > > 2. Accessibility (ie, Caribou replacement) > > > > Replacement? Trying to get Caribou properly working on the shell is not an > > option? It was already discarded? (note that I'm not saying that the shell OSK > > should be Caribou, I'm talking about a11y aspects) > > Hm... I'd thought that Caribou was not especially maintained at the moment, but > looking at git, I guess that's not true. > > The big issue is that X windows can't be used in the overview, so the keyboard > needs to be clutter-based and in-process (which in turn means it has to be > either C or JavaScript because of how GObject bindings work). > Caribou is written in Python and uses GObject introspection. I think that if only Javascript or C are supported then it is a problem and an important restriction in GNOME Shell, not in Caribou and we should fix this on the shell's side.
It's not a restriction in gnome-shell, it's a restriction in GObject; only a single garbage-collected runtime can hold a reference on a GObject at one time, or it will be leaked forever, and there is no easy fix for this.
(In reply to comment #7) > (In reply to comment #4) > > > 2. Accessibility (ie, Caribou replacement) > > > > Replacement? Trying to get Caribou properly working on the shell is not an > > option? It was already discarded? (note that I'm not saying that the shell OSK > > should be Caribou, I'm talking about a11y aspects) > > Hm... I'd thought that Caribou was not especially maintained at the moment, but > looking at git, I guess that's not true. Well, it is true that there isn't a continuous work on Caribou, but Eitan is able to book some time slots to work on it. > The big issue is that X windows can't be used in the overview, so the keyboard > needs to be clutter-based and in-process (which in turn means it has to be > either C or JavaScript because of how GObject bindings work). Ok, thanks for the explanation. It is true that it makes complex to reuse Caribou code here, and more attractive eekboard (as it is C). Anyway it is curious that eekboard claims to be "designed to be easily integrated into <skip> GNOME Accessibility" but it was never mentioned on the accessibility mailing lists. BTW, and related to the language chosen, one of the reasons to write Caribou, and discard GOK, was that it is not planned C bindings for at-spi2, at least for 3.0, just python ones. AFAIK, pyatspi2 are also based on gobject introspection, and there is a kind of internal C library for that, but it is not in a shape to became a full at-spi2 C-binding library. That would help, but there isn't any plans to do that. So this also means that this new built-in shell-osk couldn't use this (although it is not totally required, AFAIK, it is planned to get Caribou working without pyatspi, although it would miss some features) BTW2, after a quick check on eekboard, it has a optional dependency with CSPI-1.0. CSPI has been deprecated for years. Although it is an optional dependency, it shouldn't use it. If we want to really switch to at-spi2 we should really forget at-spi for once. > It is possible that this could be implemented via D-Bus like some of the status > area icons, with the "business logic" happening out of process in caribou, and > the UI happening in the shell, but I'm not sure that makes as much sense here > as with the status icons. And in the same way, it seems a lot of work on the Caribou side. Not sure if Eitan would have enough time to review it, and in the same way it seems that eekboard has already solved that (at least it talks about client-server-DBUS stuff) > Also, I haven't had a chance to dive into the caribou source yet, so I don't > have a good sense of everything it does. Same here (including eekboard, just doing a quick review right now).
Hi! I will be focusing primarily on Caribou in the next month. I was going to write somewhere about the work once I had more to show for it, but this is a good place to lay out a short-term road map. A lot of the stuff coming up on this thread has been on my mind too. The work that has gone into Caribou in recent months under my watch has been primarily platform updates, GTK3 and GSettings. Now that we are there, we could focus on broader goals. Configuration ------------- Currently Caribou has its own settings that are independent from the system keyboard settings. A tighter coupling with XKB is needed so that Caribou respects the XKB keyboard states, for example keyboard layouts selected in the shell should be reflected in Caribou. The whole bunch of other keyboard a11y features such as slow and bounce keys should also be honored by Caribou. The settings that Caribou should expose in its UI should be exclusive to an OSK, such as presentation and scanning support. Layouts ------- Currently Caribou has a handful of "layout" files that are JSON or XML keyboard geometries with keys and their associated symbols, and levels (which are misnamed as "layouts"). This will not scale as new languages are introduced. And like I said above, it does not integrate with XKB settings, and the user is forced to choose layouts in the Caribou UI. The solution for this is to separate geometries from layouts. Originally I was looking into XKB geometries, but they are problematic since they seem to be under-utilized and not well maintained. There also seems to be no magic in place for distributing 3rd party XKB geometries. What I would like to introduce in Caribou is a geometry file format similar to the layout format we have now, but that uses XKB symbolic key names, and respects XKB's current keyboard state (level and group). In the configuration of Caribou, the user would choose between 3 or 4 keyboard size geometries (eg. "small", "compact", "full"). The larger the keyboard the better language support, at least initially until we get more contributions. An addition to this format would be scanning hints, where each key could be associated with a scanning group and have an optional scanning order index. UI Separation (dbusification) ----------------------------- I have been pushing for a while to have the keyboard UI bits as trivialized as possible. Architecturally we are there, a year ago I had a Clutter-based caribou keyboard demo. A goal that I think would be worthy would be to further simplify the UI part of Caribou where there could be a core Python DBus service (which could one day be replaced with C) that would aggregate relevant AT-SPI events and info, like focus events and text widget and cursor bounding boxes (and as Dan suggested, hinting for the type of text entry). Another bit (that I already started working on), is an introspectable C library that would provide all the XKB functionality and events an OSK needs. The end-goal of all this would be to have a simple in-process JS keyboard in GNOME Shell that would take advantage of the DBus service and the C library. And do other nice stuff like you guys mentioned: moving (and maybe zooming) text entry widgets to be above the keyboard, this would be the opposite of the current behavior where the keyboard binds itself to the text widget. And use the bottom left corner as a hot corner for bringing up the keyboard. This division would also not too tightly couple GNOME Shell with caribou and would allow other shells and environments to use Caribou with other UI implementations. Broader scanning support ------------------------ This isn't directly OSK related, but having Caribou provide keyboard scanning support is only 20% of creating a usable switch user interface. Caribou will somehow need to scan all visual UI elements on the screen. The core DBus service mentioned above will probably have a lot to do with that.
I propose, once more, to take a look at Maliit [1] (In reply to comment #11) > Layouts > ------- > > Currently Caribou has a handful of "layout" files that are JSON or XML keyboard > geometries with keys and their associated symbols, and levels (which are > misnamed as "layouts"). This will not scale as new languages are introduced. > And like I said above, it does not integrate with XKB settings, and the user is > forced to choose layouts in the Caribou UI. The solution for this is to > separate geometries from layouts. Originally I was looking into XKB geometries, > but they are problematic since they seem to be under-utilized and not well > maintained. There also seems to be no magic in place for distributing 3rd party > XKB geometries. Maliit has that separation already [2], [3] > UI Separation (dbusification) > ----------------------------- > ... > > This division would also not too tightly couple GNOME Shell with caribou and > would allow other shells and environments to use Caribou with other UI > implementations. Maliit has that separation already. It's a server process which can load any kind of on-screen keyboard plugin (as long as you manage to draw your stuff inside the paint event of QWidget). That way, we can potentially integrate with any toolkit we want. [1] http://wiki.meego.com/Maliit [2] http://meego.gitorious.org/meegotouch/meegotouch-inputmethodkeyboard/trees/master/m-keyboard/layouts [3] http://meego.gitorious.org/meegotouch/meegotouch-inputmethodkeyboard/blobs/master/m-keyboard/theme/libmeego-keyboard.css
(In reply to comment #12) > I propose, once more, to take a look at Maliit [1] > > UI Separation (dbusification) > > ----------------------------- > > ... > > > > This division would also not too tightly couple GNOME Shell with caribou and > > would allow other shells and environments to use Caribou with other UI > > implementations. > > Maliit has that separation already. It's a server process which can load any > kind of on-screen keyboard plugin (as long as you manage to draw your stuff > inside the paint event of QWidget). That way, we can potentially integrate with > any toolkit we want. "use X with other UI implementations" and "as long as you manage to draw your stuff inside the paint event of QWidget" seems a contradiction to me. Could you elaborate that?
In Maliit, each plugin gets a QWidget assigned to it, that's the least common denominator. If you check the clutter-qt example [1] you will see that it uses QWidget only as the required interface to integrate with Qt, and uses clutter straight in the implementation itself. [1] http://git.clutter-project.org/clutter-qt/tree/examples/testqt.h
I suspect the worry is more about application and UI toolkit support. Maliit has a DBus API that is used to allow any application toolkit to be supported. Currently (various degrees of) work has been done for Meegotouch, Qt+Qml, Qt, Gtk+ and Clutter. So basically all the common toolkits. Regarding plugins (on-screen keyboard is an input method plugin in Maliit jargon): By using the same approach as Clutter-Qt one should be able to do a input method plugin in Gtk+ or other toolkits as well.
(In reply to comment #6) > That would need to include zooming for things like web > pages, possibly re-centering. Zooming is not an aspect of "typing with an onscreen keyboard", it's an aspect of "browsing full-sized web pages on a tiny screen". The iPhone zooms; the iPad (IIRC) only pans/scrolls. (In reply to comment #12) > I propose, once more, to take a look at Maliit [1] If it pulls in qt as a dependency then that's a probably-unsurmountable strike against it. (Because we don't want to gratuitously add gigantic dependencies that are nearly-unused, not because of NIH.)
The server library does depend on Qt, however applications using it, or the environment it is used in (here Gnome Shell) of course does not need to. Since there is no direct dependency between Maliit and the environment; and the target usescases does not cover a majority of Gnome users, I do not see why the Qt dependency should be a big problem. It is not as if Gnome would in all cases (also for those who don't care about this feature) depend on Qt.
(In reply to comment #17) > The server library does depend on Qt, however applications using it, or the > environment it is used in (here Gnome Shell) of course does not need to. Are you sure? Because taking into account this comment from Michael Hasselmann: (In reply to comment #14) > In Maliit, each plugin gets a QWidget assigned to it, that's the least common > denominator. If you check the clutter-qt example [1] you will see that it uses > QWidget only as the required interface to integrate with Qt, and uses clutter > straight in the implementation itself. It seems that if GNOME Shell wants to provide a Maliit plugin it requires a QWidget assigned. AFAIU, that means adding Qt "stuff" on GNOME Shell. And in the same way, as the server library depends on Qt, and in order to work a OSK on GNOME Shell, this library should be running, that means that the library would be running on the GNOME environment. Qt is not a module or external dependency of GNOME. So justify adding this just to have a OSK on GNOME Shell seems, IMHO, an overkill.
(In reply to comment #18) > (In reply to comment #17) > > The server library does depend on Qt, however applications using it, or the > > environment it is used in (here Gnome Shell) of course does not need to. > > Are you sure? Because taking into account this comment from Michael Hasselmann: > > (In reply to comment #14) > > In Maliit, each plugin gets a QWidget assigned to it, that's the least common > > denominator. If you check the clutter-qt example [1] you will see that it uses > > QWidget only as the required interface to integrate with Qt, and uses clutter > > straight in the implementation itself. > > It seems that if GNOME Shell wants to provide a Maliit plugin it requires a > QWidget assigned. AFAIU, that means adding Qt "stuff" on GNOME Shell. Not really. The implementation is decoupled. Only the server has a strict dependency on Qt. Application-side IM context plugins don't need Qt in any way. There is already a GTK+ IM context plugin that communicates with Maalit and it's pure GTK+ (WIP in https://gitorious.org/~csaavedra/meegotouch-inputmethodbridges/meegotouch-inputmethodbridges-update). But in any case, an external dependency in Qt is just far from ideal, even if only the server needs it. It would still mean shipping Qt by default for any distribution trying to use GNOME, and solely for this purpose. I understand that maalit has a very good architecture but unfortunately the Qt dependency is a no-go I believe.
> I understand that maalit has a very good architecture but unfortunately the Qt dependency is a no-go I believe. I think we should see this as making it possible to use the only open source on-screen keyboard that is truly capable of working properly. That seems very worthwhile. Replacing the Qt bits with GTK+ bits is doable, and could be encouraged, though I doubt anyone will really care enough to do it in practice.
(In reply to comment #20) > I think we should see this as making it possible to use the only open source > on-screen keyboard that is truly capable of working properly. This isn't Hard AI. It is entirely possible for us to make another open source on-screen keyboard that works. If GNOME's on-screen keyboard requires qt, then GNOME live images will end up being built without on-screen keyboard support, because builders aren't going to want to drop 50M worth of apps to make room to pull in qt.
(In reply to comment #21) > (In reply to comment #20) > > I think we should see this as making it possible to use the only open source > > on-screen keyboard that is truly capable of working properly. > > This isn't Hard AI. It is entirely possible for us to make another open source > on-screen keyboard that works. I think you underestimate the requirements and the work. Maalit allows for input methods that are considered standard requirements, such as phonetic asian input (Bopomofo, etc) and stroke-based input, along with prediction and error-correction for all languages/scripts. And the Maalit people have worked very hard on performance so that it's all actually responsive enough for real world use. There's a reason that several people are working hard on it over a long period of time. Nevertheless, yes, you should feel free to reimplement the Qt part, and to prove that it's easy. But there's no reason not to use the existing architecture and interfaces that don't actually involve Qt, with the Qt implementation as the placeholder.
(In reply to comment #22) > I think you underestimate the requirements and the work. Maalit allows for > input methods that are considered standard requirements, such as phonetic asian > input (Bopomofo, etc) and stroke-based input, along with prediction and > error-correction for all languages/scripts. AFAICT, Maalit does not actually do most of those things *now*, they're just planned for the future. So we've got: PRO: - lots of code already exists - many people working on it - good plans for future CON: - pulls in 50M of dependencies - has Meego's look and feel (widgets, gestures, etc), not shell's - future plans are only relevant if Nokia sticks with Meego... Anyway, Owen agrees that the qt dep makes this a non-starter. But if people want to argue further in favor of using Maalit, please take it gnome-shell-list.
OK, so based on Eitan's blog posts (http://monotonous.org/2011/04/26/caribou-week-2/ etc), it seems like Caribou is becoming more of a usable solution, for both a11y and tablet cases. So... - add the relevant devel branch of caribou to gnome-shell's jhbuild - start working on JS+clutter keyboard renderer - figure out what needs to be done in caribou to make it work in non-a11y mode as well eekboard may have relevant code
(In reply to comment #23) > CON: > - future plans are only relevant if Nokia sticks with Meego... That's a non-argument, boiling down to a generic "future plans for product $foo are only relevant if the developers of $foo keep their interest in $foo".
> - has Meego's look and feel (widgets, gestures, etc), not shell's How is http://www.youtube.com/watch?v=l_7f3Lxd4_I not Shell's look and feel?
(In reply to comment #25) > (In reply to comment #23) > > CON: > > - future plans are only relevant if Nokia sticks with Meego... > > That's a non-argument, boiling down to a generic "future plans for product $foo > are only relevant if the developers of $foo keep their interest in $foo". If you rely on a library, or service of that size, you would hope that it would stay maintained. If we wanted to maintain something like that ourselves, we'd probably want to rewrite it (in a language, and using frameworks that we know, rather than something wholly different). So it's relevant, even though it shouldn't be a concern. (In reply to comment #26) > > - has Meego's look and feel (widgets, gestures, etc), not shell's > > How is http://www.youtube.com/watch?v=l_7f3Lxd4_I not Shell's look and feel? It doesn't use the shell's widget set, so it won't match look and feel exactly. I'll add another CON: Doesn't work in the shell's overview. If the service is awesome, we might be able to build on top of it in the shell, and potentially replace the Qt core of the daemon later on. I'm still concerned about Caribou's X event handling, and how it would integrate with gnome-shell itself, which is necessary if we want it to work in the overview.
(In reply to comment #27) > [...] if we want it to work in the overview. Given that search is the preferred way of getting to applications/documents/places (and people in the future), I don't think that supporting the shell's overview is optional.
(In reply to comment #27) > I'll add another CON: Doesn't work in the shell's overview. It has pluggable UIs, to some degree; the mockup linked above wouldn't work in the overview, but I believe that is fixable. > I'm still concerned about Caribou's X event handling, and how it would > integrate with gnome-shell itself, which is necessary if we want it to work in > the overview. See the blog post linked above; Caribou master has a view/controller split. It would work basically like the bluetooth/network/power status icons.
> It doesn't use the shell's widget set, so it won't match look and feel exactly. It was able to match the provided mockup pretty well with a couple hours of work, so I do not see why it would not be able to match the theme/style used by the shell toolkit widgets. There is nothing that prevents you from writing your own keyboard plugin for Maliit that uses the shell toolkit (using ClutterQt), if you think that is a must. > I'll add another CON: Doesn't work in the shell's overview. Neither does any other existing solution, so I'd hardly call it a CON. :) Fixing this requires an input context to be implemented for the shell toolkit*, like we already have input contexts for Qt and Gtk. In addition some cooperation between the window manager and Maliit would probably be required to allow the input method window to stay on top of the shell overview. The code used in the Gtk+ input context is split into two parts, the Gtk+ IC module code and a generic wrapper library (C/GObject) around the IC DBus API. The generic part could be used to implement a shell toolkit input context. * If the shell toolkit uses clutter-imcontext, one inputcontext would be able to support both Clutter and Shell Toolkit.
Ahem: > Anyway, Owen agrees that the qt dep makes this a non-starter. But if people > want to argue further in favor of using Maalit, please take it to > gnome-shell-list. Please do not clutter up this bug with further discussion.
> 1. Tablets and other large form-factor touch screens. (Not phones.) > > - To be easy to use, the buttons will have to be largish, > making the keyboard as a whole take up a good chunk of the > screen. We will probably want it to stay in place, at the > bottom of the screen, as described above. As someone who has used a tablet for *drawing*, I would strongly recommend that the keyboard not take up the screen full-time. I'd much prefer the drawing canvas take up nearly all the screen most of the time, and have the keyboard only pop up when needed.
Right, by "stay in place", I meant "always be in the same place when it's visible", as opposed to the caribou keyboard, which moves around the screen (when it's visible) to always be near the window where you're typing.
So is there a list of functionalities that current versions of caribou, eekboard, maalit, onboard could be compared with, to get this to a rational level again?
*** Bug 649825 has been marked as a duplicate of this bug. ***
Another data point: Adaptxt http://www.openadaptxt.org/ make an open source (EPL licensed) predictive text engine including a number of user interfaces (including an onscreen keyboard). Perhaps Caribou could make use of that, or alternatively it could be considered for inclusion in GNOME? I know that the Adaptxt guys are happy, even eager, to have a closer association with GNOME. I don't know to what extent the EPL license would come into consideration when thinking about closer integration. Cheers, Dave.
FYI, As I mentioned to Dave Neary on #release-team, some people already made some prediction tests with Caribou, using Presage as the prediction engine, and creating a new input method: http://www.joaquimrocha.com/2010/03/03/text-prediction-on-gnome/ http://www.joaquimrocha.com/2010/04/05/caribou-and-text-predictor-input-mode/ http://dpellicer.warp.es/?p=6 So just another option to take into account. (BTW, not sure if prediction fits here or it would be required a new bug)
Configuration, etc, questions for the designers: - (see also bug 652646) - Even if we have clever heuristics to detect when no physical keyboard is present, we still want some way somewhere to be able to enable the OSK, for eg, when your keyboard is slightly broken ("th ky btwn 'w' and 'r' dosn't work!"), or is for some reason difficult to access. - Currently there are some configuration options associated with the keyboard (in particular, (1) whether to use an iPad-esque "simple" keyboard or a full keyboard with function keys and Alt and PrintScreen, etc; and (2) whether to keep the keyboard at the bottom of the screen or let it float). We may manage to deal with those via clever heuristics too, but if not, we need somewhere to adjust those - There probably needs to be some way to explicitly pop up the keyboard, for situations where you want to type something, but either (a) you don't have a text entry focused, or (b) you have a text entry focused in some app that we can't detect text focus in. (At the moment, that's "any non-gtk app", but that will probably change.) It would probably make sense to have some of the OSK stuff in the same status icon/menu as the keyboard layout stuff, but that raises various questions like when exactly would this combined menu be shown, and what would it look like.
I think the way to enable the osk is: * Entry focused without text input device connected * Gesture * Universal Access settings I don't think that we want a floating keyboard ever. We don't want to manage yet another window nevermind a "window" provided by the OS chrome. This doesn't necessarily mean that it has to take up the entire width of the screen either. Just that it shouldn't be managed or positioned manually.
Created attachment 193891 [details] [review] layout: add panelBox and trayBox Have LayoutManager automatically deal with sizing and positioning boxes for the panel and messageTray relative to the monitors. Also, now that LayoutManager knows exactly where and how tall the panel and tray are, have it manage the pointer barriers as well.
Created attachment 193892 [details] [review] Updated on-screen-keyboard patches (squashed) === needs a real commit message, and this is not the final form. but if we want to get it in for feature freeze, then we should commit it now, and I'll continue cleaning up the code post-freeze. So, don't review the keyboard.js parts, just the other integration stuff. With this patch, the keyboard is only used if you set org.gnome.shell.keyboard show-keyboard to true, and this setting is not exposed in the UI anywhere. So, even if it's not fully working, it shouldn't break anyone who isn't trying to use it if we commit it now.
How can users know if the current code is merged? For example the alphas of ubuntu 11.10 have gnome-shell 3.1.5 installable. At what shell version will I be able to test the keyboard? If there is still no clear schedule please excuse my premature question.
This code will be in 3.2.
(In reply to comment #42) > How can users know if the current code is merged? in general, if a patch is committed, its status would be changed from "none" to "committed" in bugzilla
Review of attachment 193891 [details] [review]: Mostly looks good to me, but there are some aspects of the handling of the message tray at the bottom which could use some commenting. ::: js/ui/layout.js @@ +47,3 @@ + this.trayBox = new St.BoxLayout({ name: 'trayBox' }); + this.trayBox.connect('allocation-changed', + Lang.bind(this, this._updateTrayBarrier)); Hmm: We attach to allocation-changed on this actor, but read Main.messageTray.actor.height. Is this an ordering things? The height of trayBox is always 0, because it's child is fixed position, but we're counting on allocation-changed being emitted when the child changes size? Why do we have trayBox and bottomBox? Basically, there's some oddness here, and some comments elucidating what is going on would be useful.
Created attachment 195011 [details] [review] Comments for layout patch Thought I'd try to help out Dan based on what I remember from our discussions on containers for the keyboard/tray. Hope it helps!
(In reply to comment #45) > We attach to allocation-changed on this actor, but read > Main.messageTray.actor.height. Is this an ordering things? The height of > trayBox is always 0, because it's child is fixed position, but we're counting > on allocation-changed being emitted when the child changes size? Basically, yes. > Why do we have trayBox and bottomBox? Hm... because of slightly-bad refactoring. I should have left bottomBox out until the second patch. The idea is that if you think about the 4 states of tray visible/hidden and keyboard visible/hidden, you realize that the tray is not actually attached to the bottom of the screen, it's attached to the top of the keyboard. So bottomBox exists to hold trayBox and keyboardBox.
OK, I pushed this with a slightly reworked version of the layout stuff, to not add bottomBox until the keyboard patch, and added a comment there as well. As it currently stands, the keyboard is still enabled by a hidden gsettings key; I tried switching it over to using the "real" key, but at the moment at least, enabling that key adds at-spi to the xsettings gtk modules list, which causes everything to break a lot.
Created attachment 195094 [details] [review] keyboard: switch to using the correct gsettings key for enable/disable
I haven't been able to reproduce the alleged brokenness due to a11y being turned on; neither when using the osk on the login screen nor in the session.
(In reply to comment #50) > I haven't been able to reproduce the alleged brokenness due to a11y being > turned on; neither when using the osk on the login screen nor in the session. Right, maybe I didn't make it clear how to reproduce that problem. I don't want to have 2 settings in the control-center, one for gnome-shell, and one for the fallback Caribou. Make the necessary changes in js/ui/keyboard.js for it to use: org.gnome.desktop.a11y.applications screen-keyboard-enabled instead of: org.gnome.shell.keyboard show-keyboard And pain ensues (because Caribou, the app, keys off that same key to start itself up[1]). We could certainly remove that autostart, and make gnome-settings-daemon launch Caribou when in fallback mode (which it would need to detect). [1]: http://git.gnome.org/browse/caribou/tree/data/caribou-autostart.desktop.in
(In reply to comment #51) > Make the necessary changes in js/ui/keyboard.js for it to use: > org.gnome.desktop.a11y.applications screen-keyboard-enabled > instead of: > org.gnome.shell.keyboard show-keyboard Which is what Dan's last patch does. I guess I should be in bed now :)
Don't know were the GSettings discussion originated In the scope of the shell I would keep out the "Caribou keyboard" (ie. Antler) out all together and not have it as an option. Maybe that autostart file should go away, I forgot it was there. Since Dan seems to be pushing it in a direction where we would not use AT-SPI, we are not dependant on the a11y gsettings dance anymore. And the Caribou daemon would not need to run. After writing the last two sentences I realized how confusing the different pieces in the Caribou module are. Sorry. If there needs to be some clarification, I'll follow-up.
(In reply to comment #52) > > Which is what Dan's last patch does. I guess I should be in bed now :) Good night !
(In reply to comment #50) > I haven't been able to reproduce the alleged brokenness due to a11y being > turned on; neither when using the osk on the login screen nor in the session. yeah, as discussed on IRC, my problem was caused by bad gconf settings causing the corba at-spi to be loaded, and Ray can't reproduce his bug any more. (In reply to comment #53) > In the scope of the shell I would keep out the "Caribou keyboard" (ie. Antler) > out all together and not have it as an option. Maybe that autostart file should > go away, I forgot it was there. except that we still want antler to run in fallback mode. > Since Dan seems to be pushing it in a direction where we would not use AT-SPI, > we are not dependant on the a11y gsettings dance anymore. But without at-spi we don't get keyboard support in non-gtk apps. As long as at-spi doesn't kill puppies, there should be no problem with running it... I need to upgrade to F16 so I can test with current gtk3 and caribou more easily.
(In reply to comment #55) > except that we still want antler to run in fallback mode. Which *might* currently not work if bug 658533 is valid.
(In reply to comment #51) > (In reply to comment #50) > We could certainly remove that autostart, and make gnome-settings-daemon launch > Caribou when in fallback mode (which it would need to detect). > > [1]: http://git.gnome.org/browse/caribou/tree/data/caribou-autostart.desktop.in Is somebody working on a patch to that end, then ? Would be good to have this sorted out by .92
(In reply to comment #57) > Is somebody working on a patch to that end, then ? yes
*** Bug 659192 has been marked as a duplicate of this bug. ***
Attachment 195094 [details] pushed as 554ad4e - keyboard: switch to using the correct gsettings key for enable/disable latest caribou is required to avoid crashes closing this bug; further issues should be filed as separate bugs
+ let screenKeyboard = this._buildItem(_("Screen Keyboard"), APPLICATIONS_SCHEMA, This broke the string freeze. Please revert or ask for an exception on gnome-i18n@.
Sorry to speak up in a closed bug, but I wonder if there is any documentation available about this. I have lots of questions such as: - What is the second key in the bottom right corner (the one with the 3 dots) supposed to do? - Why are some keys highlighted? - What about special keys like TAB or arrow keys? - Why doesn't the keyboard show up in a terminal automatically? - What is the proper component to file bugs against if I have problems with the keyboard?
(In reply to comment #62) > Sorry to speak up in a closed bug, but I wonder if there is any documentation > available about this. I have lots of questions gnome-shell-list@gnome.org would be a better place for these > - What is the second key in the bottom right corner (the one with the 3 dots) > supposed to do? there's not supposed to be a key with three dots. If there is, it means the actual label is getting ellipsized, which would be a bug. > - Why are some keys highlighted? Not sure what you mean. There shouldn't be any highlighted keys. > - What about special keys like TAB or arrow keys? They may be supported later. > - Why doesn't the keyboard show up in a terminal automatically? Because the terminal isn't a GtkEntry or a GtkTextView, and the code doesn't deal with anything else yet. > - What is the proper component to file bugs against if I have problems with the > keyboard? gnome-shell/keyboard