GNOME Bugzilla – Bug 709999
Wayland: Mouse event synthesis is not working
Last modified: 2021-07-05 10:45:31 UTC
Steps to reproduce: 1. Launch a python console 2. Try the following: >>> import pyatspi >>> pyatspi.Registry.generateMouseEvent(400, 400, "abs") Expected results: The mouse pointer would move to coordinates 400, 400. Actual results: The mouse pointer does not move. Impact: Orca's flat review feature includes commands to route the mouse pointer and synthesize clicks. This is needed in order to provide access to applications (including web apps) which are not fully keyboard accessible and is accomplished via AT-SPI2. These Orca commands cannot be used in Wayland.
confirmed. I was testing gnome 3.16 with wayland the other day and mouse functions indeed did not work. Although, flat review did. Adding keyword accessibility to help devs find this bug.
Same answer here as for bug 710012: to keep this working without major changes to the a11y architecture, registryd needs a wayland implementation of deviceeventcontroller that talks to a private interface of the compositor.
confirmed it still happens at least on 3.22
confirmed on 3.28.2
We have very recently been looking into very similar problem in order to make dogtail be able to do input actions under gnome wayland session too. Problem is that the at-spi registryd is tied to Xtest which is obviously of no use with real wayland clients. So we have workarounded that by taking advantage of the new gnome-remote-desktop and screen-cast dbus interfaces that provide the necessary api allowing to do mouse as well as keyboard input. Second issue is that the at-spi only return 'local' window coords for any object, which would need to be translated to global ones before this API could be used. And so Olivier Fourdan was kind to add to the existing interface available in mutter, to a 'RecordMonitor' (needs global coords) also a 'RecordWindow' interface that when told what window in question we want to input on - it will take the local coords, translating to global ones internally in mutter (so that we never get to know the global ones really - which was demanded in order to keep the Wayland's security aspects.) Additionally, in order to be able to identify particular window in question, he also added GetWindows function to the gnome-shell introspect interface. And to make use of all these interfaces easy to use Olivier also wrote the 'gnome-ponytail-dameon' - a dbus daemon that bridges all this together, so the 'user' needs to only work with its dbus interface directly. And since this was originally done 'for' dogtail, gnome-ponytail-damon also installs a python module called 'ponytail' that contains methods that can be used from python direcly without having to care for any actual dbus calls and that are designed so they are very similar to the original registryd functions (see https://gitlab.gnome.org/ofourdan/gnome-ponytail-daemon/blob/master/ponytail/ponytail.py) Although originally done for dogtail, ponytail makes no ties to that and can be used on its own by any tool in need (orca?). Furthermore, we were able to push all these changes to the upstream of mutter and gnome-shell (I believe from 3.32?), so this is a go for anyone, no special deps needed except for the gnome-ponytail-daemon. Take a look at https://gitlab.gnome.org/ofourdan/gnome-ponytail-daemon/, comes with good description of this solution as well as examples. So replacing the registryd calls for ponytail ones is made rather easy - another part is to always be able to determine the window ID we need to work with, which depends on particular tools internal structure. I.e we were able to do this in dogtail in such a way that each UI Node (accessible) knows it's window ID by traversing up the at-spi tree, while also checking up against what the GetWindows method provides (window titles equal to those that this gnome-shell introspect methods returns) (i.e. you can see the changes done in dogtail here https://gitlab.com/dogtail/dogtail/tree/wayland, still a bit messy draft looking code but confirmed to work and have restored nearly all original functionality - mainly look at rawinput module and Node.WindowId method from tree module)
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org. As part of that, we are mass-closing older open tickets in bugzilla.gnome.org which have not seen updates for a longer time (resources are unfortunately quite limited so not every ticket can get handled). If you can still reproduce the situation described in this ticket in a recent and supported software version, then please follow https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines and create a new ticket at https://gitlab.gnome.org/GNOME/at-spi2-core/-/issues/ Thank you for your understanding and your help.