After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 319735 - gok works poorly as an input mechanism on tablet PCs
gok works poorly as an input mechanism on tablet PCs
Product: gok
Classification: Deprecated
Component: general
Other Linux
: Normal enhancement
: ---
Assigned To: David Bolter
David Bolter
Depends on:
Blocks: 323648 518864 519313
Reported: 2005-10-25 13:44 UTC by Matthew Garrett
Modified: 2011-10-14 10:58 UTC
See Also:
GNOME target: ---
GNOME version: 2.11/2.12

Description Matthew Garrett 2005-10-25 13:44:06 UTC
A typical tablet PC X configuration has the following:

1) A PS/2 core pointer
2) A synaptics pad with "SendCoreEvents" "true"
3) A cursor device for the tablet, with "SendCoreEvents"
4) A stylus device for the tablet, with "SendCoreEvents"
5) An eraser device for the tablet, with "SendCoreEvents"

Ideally, gok would be usable with the cursor device without preventing it from
being used as the core pointer (tablets aren't very useful otherwise). At the
moment there doesn't seem to be any way of configuring that.
Comment 1 David Bolter 2005-11-24 15:15:25 UTC
Matthew, thanks for this bug report. It would be good to have a corepointer mode
(perhaps a flag) where devices were not automagically unhooked. I'm not 100%
sure  gok can't operate this way currently... but this area needs to be looked
at and clarified a bit. (Note we are also looking into alternatives to using
XINPUT extensions for adaptive hardware input) 

Perhaps we should have a flag:
-c, --c      Use the core mouse device as is.
Comment 2 bill.haneman 2005-11-24 15:23:45 UTC
Matthew, unfortunately it isn't technically possible to use GOK via the
corepointer.  Talk the the toolkit folks if you think that's wrong, nothing we
can do ATM.

David, I don't think 'using core mouse device as is' will help, as it will just
cause "hangs" and "lockouts" from the users' POV.  Apps are constantly doing
pointer grabs, which stops GOK from working.

IN any case, GOK currently allows you to "cancel" if you don't want to unhook
the core device.

Comment 3 David Bolter 2005-11-24 15:31:58 UTC
Matthew, is it apparent to you (wearing your gok user hat) that cancel does not
unhook the core device?
Comment 4 bill.haneman 2005-11-24 15:34:30 UTC
I would concede the end result of pressing 'Cancel' probably isn't apparent to
the user.
Comment 5 David Bolter 2005-11-24 15:36:47 UTC

BTW Bill, I envision the -c option being for users who can use the regular mouse
but happen to require an on-screen keyboard (for character input). We would
bypass all the hardware detection stuff.
Comment 6 bill.haneman 2005-11-24 15:39:12 UTC
Yes - though even for character input it may act broken, since more and more
file choosers and entry widgets and what-not seem to be doing grabs.  
Comment 7 Matthew Garrett 2005-11-24 15:41:20 UTC
On starting gok, regardless of whether I choose to unhook the core pointer or
not, I currently get two mouse pointers. One follows the pen around the screen
and allows me to interact with other applications - the other remains within
gok. This wouldn't be a problem except that the pointer within gok reacts even
if the screen-wide pointer is outside gok. So possibly the behaviour I want is
simply for gok not to react if the core pointer is outside the gok window - that
way, the core pointer could be used to interact with applications and the gok
pointer to interact with gok. Would this work?

(Sorry, I'm not entirely clear on how the pointers interact with gok, so I may
be talking complete nonsense here)
Comment 8 David Bolter 2005-11-24 15:42:23 UTC
I still feel like this grab behaviour ought to be end-of-life'd somehow.
Comment 9 David Bolter 2005-11-24 16:05:11 UTC
Matthew, I'm too foggy on gok's input device magic ATM to comment intelligently
other than to say I have seen the behaviour you describe. I think you idea for
solution has merit as would perhaps not even having a second (gok) pointer for
core mode.
Comment 10 bill.haneman 2005-11-24 16:40:21 UTC
Matthew: behavior you describe in #7 sounds like misconfiguration to me.  If
you're seeing two pointers moving "at once", you have a misconfigured Xserver
and/or GOK.
Comment 11 Matthew Garrett 2005-11-24 16:55:26 UTC
In X, I have:

Section "ServerLayout"
        Identifier      "Default Layout"
        Screen          "Default Screen"
        InputDevice     "Generic Keyboard"
        InputDevice     "Configured Mouse"
        InputDevice     "Synaptics Touchpad"
        InputDevice    "cursor"    "SendCoreEvents"
        InputDevice    "stylus"    "SendCoreEvents"
        InputDevice    "eraser"    "SendCoreEvents"

In "actions" in gok, I have event source set to core pointer. Moving the
touchpad on the system results in the "gok pointer" (a small cross) moving
within the gok window. Moving the pen on the screen results in both the "gok
pointer" and the normal system cursor moving. The X setup is the recommended
configuration for wacom-based tablet systems, and "cursor", "stylus" and
"eraser" are the three different wacom input devices.
Comment 12 bill.haneman 2005-11-24 17:06:00 UTC
The problem is that these 'different input devices' are connected to XInput as
extended devices, but are still sending core events.  THis is a major problem
for GOK, since GOK sees these as non-core input devices (GOK can't tell that
SendCoreEvents is set).

Comment 13 Matthew Garrett 2005-11-24 17:14:40 UTC
Unfortunately, I can't see any workable strategy that doesn't involve them
sending core events - it has to be possible to drive the system cursor with the
stylus. Is it possible for gok to tell that the system cursor (I've probably got
the terminology wrong here - is that in fact the core pointer?) is outside the
gok window, and if so not update the gok pointer?
Comment 14 bill.haneman 2005-11-24 17:17:58 UTC
No, the GOK pointer should not be updating at all if the core pointer is being
moved.  It seems that GOK is getting events from the XInput device channel in
this case, and it cannot ignore such events and still work in its recommended
configuration (which would be without SendCore on the XInput extended device
being used).

Comment 15 bill.haneman 2005-11-25 15:56:57 UTC
Here's the dilemma, I think, that leads to this behavior:

1) USB devices are prone to renumbering across boots;
2) GOK therefore needs to auto-detect unexpected XInput events, since otherwise
a renumbering will cause the pre-configured GOK XInput device connection to fail
without offering the user a solution.
3) however, SendCore devices are unfortunately not identified by the XInput
extension as "core" events (though CorePointer events are).

3 means that all SendCore devices will generate XInput events which GOK cannot
distinguish from case #1 above, and the code to auto-detect in #2 gets invoked,
causing GOK to receive these events as XInput events.

Because they are SendCore, although GOK is seeing them as extended devices, the
devices in question still move the core pointer, and are still subject to the
pointer grab problems as a result of application and toolkit pointer grabs.

I don't see an obvious solution here, other than (configurably) turning off the
auto-detection of XInput devices by GOK.  
Comment 16 korn 2005-11-27 07:43:42 UTC
Stepping outside of the current and immediate X constraints, would: (a) libusb
allow us to solve this problem? [could Matthew still get his events that way?]
or (b) XEvIE, especially if extended to support multiple clients?
Comment 17 bill.haneman 2005-11-27 11:36:05 UTC
verily if we could but cast aside those shackles and GOK freely...

There is some serious question as to whether libusb can be used with devices
which are serving as core pointer and/or recognized already as HIDs.  We suspect

XEvie, if extended to multiple clients, could do it for us.
Comment 18 Matthew Garrett 2005-11-27 12:49:49 UTC
Tablet PCs generally (actually, pretty much exclusively) have devices connected
via a serial interface, not USB. We can probably make guarantees about the
device that they're connected to, but I'm not sure if that's useful.
Comment 19 Ben Konrath 2005-11-27 21:12:43 UTC
In response to comment #17, libusb can be used with devices that have been
recognized HID by detaching the device from the kernel driver. Libusb provides a
function to do this but it seems to be Linux specific right now. Once the device
is detached from the kernel driver, it no longer serves as the core pointer
because the underlying entry in /dev is no longer present. 

I'm still not exactly sure if this will help with the problem at hand.
Comment 20 David Bolter 2005-11-28 14:10:00 UTC
A bit of an aside:
WRT the whole issue of having a non-core gok mouse to avoid mouse grabs... I
wonder if we could trap mouse grab calls and react at that stage: 
- not allow the mouse grab?
- unhook from core at that point, re-hooking after the grab?
I doubt this is possible currently, but it might be a future compromise (with a
bit of work).
Comment 21 bill.haneman 2005-11-28 15:01:01 UTC
There's no way to trap mouse grab calls, unless you're the X server.  Client
applications can't do that.  It makes more sense to extend XEvie, since allowing
client applications to snoop calls from other clients is very troublesome for
lots of reasons (i.e. it breaks the X model, introduces security issues, causes
performance roundtrips, etc. etc.)
Comment 22 David Bolter 2005-11-28 15:14:13 UTC
Yeah I hesitated to move the conversation in this direction because the
libusb/xevie/etc work is important. I notice:
"Option "AllowDeactivateGrabs" "boolean"
    This option enables the use of the Ctrl+Alt+Keypad-Divide key sequence to
deactivate any active keyboard and mouse grabs. Default: off. "

I'd like to see this deactivation assigned to gestures.  Not now, but something
to think about?
Comment 23 bill.haneman 2005-11-28 15:18:45 UTC
Bear in mind that this deactivation is for "active" grabs, i.e. only works when
a grab is already in effect.  The other grabs that applications do are "passive
grabs", which basically means "pending grabs", which take effect (i.e. become
active) when a particular key combination is pressed.

It's possible that GOK could be made to programmatically make this call if it
stops receiving valuator input, I am not sure.
Comment 24 David Bolter 2005-11-28 15:29:18 UTC
OK, I've created bug 322655 for the mouse grab discussion.
Comment 25 Calum Benson 2006-04-26 17:11:21 UTC
Apologies for spam... ensuring Sun a11y folks are cc'ed on all current accessibility bugs.
Comment 26 chris kirkpatrick 2007-05-13 13:11:13 UTC
I have essentially the same setup as Matthew (comment #11). My problem is that I cannot achieve shift, control or alt key effects using the onscreen Compose keyboard, either with the touchpad mouse pointer or with the tablet stylus. If I touch the Caps Lock button, the little triangle lights up and I can achieve capital letters, but with the other keys the little triangle doesn't light up and the button doesn't function. 

Bizarrely, if I hold down the Shift or Control button on the conventional laptop keyboard and then touch a key with either mousepointer, the shifted or controlled function of that key now works (and the little triangle in the GOK button lights up). This is not, of course, a useful way to use the onscreen keyboard!

The same problem is encountered on a conventional desktop computer using ordinary keyboard and mouse; the Control, Alt and Shift keys don't appear to function when pointed to by th eon-screen device, though their function is activated by touching the corresponding key on the main keyboard.

Setup is fairly conventional: Desktop with Mepis 32 v 6.5, GOK v 1.0.10; laptop is Toshiba Satellite R10-101 with same versions of software. Wacom stylus etc devices are found at /dev/ttyS4.

Any thoughts?
Comment 27 David Bolter 2007-05-14 00:43:27 UTC
Chris, do you have Sticky Keys set up and working? 
Comment 28 chris kirkpatrick 2007-05-19 06:47:36 UTC
As soon as it launches, I am told in a dialog box: 'GOK has enabled
sticky keys, which it needs'.

I don't know whether or not this is true, and what is its significence?

I also get these messages at the terminal:

(gok:7640): Gnome-WARNING **: Accessibility: failed to find module
'libgail-gnome' which is needed to make this application accessible
GTK Accessibility Module initialized
/dev/js0: No such file or directory
/bin/sh: /usr/bin/esd: No such file or directory
134958736 event types available
login mode = false
Word prediction dictionary contains a total of 3000 words

I have installed libgail-gnome from the repository; it doesn't solve the
problem. I cannot find esd as such in the repositories, though it
appears as a suffix to several packages; it seems to have something to
do with sound so is probably irrelevant to my problem. I presume
/dev/js0 is a joystick, which is not present in my system.

Comment 29 Akhil Laddha 2011-10-14 10:58:15 UTC
gok (GNOME on-screen keyboard) development has been stalled and it has been replaced by caribou [1]. Maintainers don't have future development plan so i am closing all the bugs as WONTFIX.