GNOME Bugzilla – Bug 545162
[a11y] Broken accessibility hierarchy
Last modified: 2008-09-15 17:22:30 UTC
We've been very excited to see the Gecko 1.9 work (see bug #484991 and bug #499744) and have recently been able to test the results of this effort. On Ubuntu Intrepid with the latest GNOME 2.23.5 bits, and with yelp built/installed from SVN trunk, we're seeing some very bad accessibility hierarchy issues. The next comment will include a test application to this bug that demonstrates the problem.
Created attachment 115432 [details] Test application to demonstrate the problem - run via "python yelp_bug.py" in an xterm This simple standalone pyatspi-based test application helps demonstrate the problem. What it does is look for caret-moved events and then tries to look upwards in the hierarchy to find the application associated with the event. When it hits the document frame, we run into hierarchy issues. This might be some odd interaction between Gecko and GTK+, but I'm not sure. Here's a sample set of steps: 1) Run yelp 2) Run this test application in an xterm: python yelp_bug.py. It's important to use an xterm and not a gnome-terminal so as to avoid getting tons of events from gnome-terminal. 3) Position the caret somewhere in the yelp content (e.g., between the 'l' and 'c' in 'Welcome to the GNOME Help Browser' -- you may need to press F7 to enable caret navigation (I can't remember if I did this or not) 4) Arrow left and right and observe the output in the xterm window 5) Position the caret in the 'Search:' text area. Type something and observe the output in the xterm window. What you will see is this: 1) The test app first lists the known accessible apps on the desktop. My looks something like this (you'll see that 'yelp' is there): [application | metacity] [application | gnome-panel] [application | gnome-volume-manager] [application | update-notifier] [application | bluetooth-applet] [application | evolution-alarm-notify] [application | tracker-applet] [application | applet.py] [application | gnome-terminal] [application | nm-applet] [application | gnome-power-manager] [application | nautilus] [application | Firefox] [application | yelp] 2) When arrowing around in the help content, you see a bad hierarchy from the 'document frame' object on up. We end up on some unknown application object, which is not good. Also note that the event.host_application seems wrong as well (it doesn't say 'yelp'): caret moved: heading event.host_application: application [application | ] event.source.getApplication(): [application | ] ancestry of event.source: +- [application | ] +- [document frame | Desktop] +- [section | ] +- [section | ] +- [heading | ] 3) When entering text in the "Search:" text area, we see goodness where we end up on the 'yelp' application. Note, however, that the event.host_application still seems suspect as above: caret moved: text event.host_application: application [application | ] event.source.getApplication(): [application | ] ancestry of event.source: +- [application | yelp] +- [frame | Desktop] +- [filler | ] +- [tool bar | ] +- [panel | ] +- [filler | ] +- [text | ] 4) Look also at the yelp hierarchy in accerciser. It seems to look good when looking down from the top accessible. So, the problem seems that it might be when looking upward from a child accessible.
Adding Ginn Chen to the CC list since he's a Gecko expert. I'm wondering if there is just some special magic that needs to be done to hook up the Gecko document frame to the GTK+ hierarchy.
I can reproduce the issue on Ubuntu 8.04. To have two accessible toolkits (gail, Gecko) working at the same time is a really tricky. Firefox works fine with native file-picker dialog and print dialog, it's a trick. When Firefox starts, it loads and inits libgail, then inits Gecko a11y toolkit and overwrites atk_get_root, atk_get_toolkit_*, then it inits atk-bridge. So you will see "Firefox" in accerciser, and print acc.getApplication() also returns "Firefox". The yelp case is similar, yelp inits gail and atk-bridge when it starts. When atk-bridge inits, it calls atk_get_root, and registers "yelp" as the application name. That's why you see "yelp" in accerciser left panel. Then yelp loads Gecko, Gecko hijacks gail atk_get_root callback. atk-bridge is already initialized, so the application name in accerciser left panel won't change. But if you call atk_get_root (acc.getApplication) to any accessible object under yelp, it will call Gecko's atk_get_root callback. I think Gecko atk_get_root callback would fail if Firefox/Thunderbird is not the host. I think we can make Gecko smarter, to check if it is actually embedded when it overwrites callbacks. But it would be really tricky. Since atk_get_toolkit_* only returns one result, and I think "Gecko" is the expected result for yelp, from Orca's point of view. So Gecko should overwrite atk_get_toolkit_* but not atk_get_root, in this case.
Thanks for the details, Ginn! Do you think there's anything yelp could do to poke and prod at Gecko in its existing form in some way to make things work out? The concern I have is that if we need to wait for changes to be made to Gecko, GNOME 2.24 is likely to go out with an inaccessible yelp (i.e., it'll be likely the modified Gecko won't be used).
More information, On Solaris, getApplication() returns [application | Firefox ]; on Ubuntu 8.04, it is [application | ]. The reason is Ubuntu ships xulrunner separately, so Gecko failed to get the brand name. Evan we can fix the name, it's not right. The Gecko app root accessible object would not have the right child count. And it's not the same accessible object we can get from doing get parent repeatedly. So, for Gecko, we need to know whether it's embedded. I don't know how to do that yet. It's unlikely to be fixed in 3.0.x. Thus, I proposed a hack for yelp to get around.
(In reply to comment #5) > More information, > On Solaris, getApplication() returns [application | Firefox ]; on Ubuntu 8.04, > it is [application | ]. > The reason is Ubuntu ships xulrunner separately, so Gecko failed to get the > brand name. Note that Orca uses the application name to determine the script class to use. Once a script instance is created from a script class, however, Orca uses the application object (not name), to determine which script instance to use. > Thus, I proposed a hack for yelp to get around. Thanks Ginn! Are you currently working on the hack?
Created attachment 116420 [details] [review] patch
(In reply to comment #6) > (In reply to comment #5) > > More information, > > On Solaris, getApplication() returns [application | Firefox ]; on Ubuntu 8.04, > > it is [application | ]. > > The reason is Ubuntu ships xulrunner separately, so Gecko failed to get the > > brand name. > > Note that Orca uses the application name to determine the script class to use. > Once a script instance is created from a script class, however, Orca uses the > application object (not name), to determine which script instance to use. "application name" doesn't mean "application object name", right? "application object name" is supposed to be localized.
(In reply to comment #8) > "application name" doesn't mean "application object name", right? > "application object name" is supposed to be localized. We use the accessible.name field of the object to determine the script to use, and we provide mappings from localized names to the script name. There unfortunately doesn't appear to be a more locale-independent way. PS - Thanks for the patch! We'll try it out.
I just did a very quick test with this patch. Using it, and making a correction in one of Orca's script files, the events now seem to be getting to the right place. :-) We'll need to write a yelp-specific script (or add to the Mozilla script). I'll work on that and let you know. But so far so good Ginn. Thanks!!!
It seems that the hierarchy is still broken though. Given this hierarchy: [...] - Split Pane - Scroll Pane - [...] - Filler - Document Frame If you ask the Filler for its first child, the Document Frame is returned; but ask the Document Frame for its parent and an accessible of ROLE_APPLICATION is returned rather than the Filler.
(In reply to comment #10) > I just did a very quick test with this patch. Using it, and making a correction > in one of Orca's script files, the events now seem to be getting to the right > place. :-) We'll need to write a yelp-specific script (or add to the Mozilla > script). I'll work on that and let you know. But so far so good Ginn. > Thanks!!! > Thanks for the testing! I think there's a little problem. Gecko a11y module might be initialized by at tools before html_close, so we should get gail_get_root earlier. I'll give another patch for fix that.
(In reply to comment #11) > It seems that the hierarchy is still broken though. Given this hierarchy: > > [...] > - Split Pane > - Scroll Pane > - [...] > - Filler > - Document Frame > > If you ask the Filler for its first child, the Document Frame is returned; but > ask the Document Frame for its parent and an accessible of ROLE_APPLICATION is > returned rather than the Filler. > Thanks for catching this. It's another bug. Maybe we can reset the parent, another hack ...
I use atk_object_set_parent() in yelp-html.cpp to reset the parent to yelp filter object. The weird thing is when I click it in accerciser, somewhere called ref_child for the Gecko App root object, then the parent of "document frame" is set back to Gecko app root object. It looks like the call is from another process, I don't know the detail yet.
(In reply to comment #14) > It looks like the call is from another process, I don't know the detail yet. > I got it, "Document Frame" has RELATION_NODE_CHILD_OF, it still directs to the Gecko App root object. When accerciser tries to get the information, ref_child is called, parent of "Document Frame" is set back again. That would be hard to fix.
Joannie, Does Orca use relationsets? Is this bug still worth fixing if we leave the issue of #c11? I think the relationset and parent of "Document Frame" is pretty hard to fix from yelp's side.
> Does Orca use relationsets? It does. But we can work around that I *think*. > Is this bug still worth fixing if we leave the issue of #c11? Well, without your patch, we can't seem to do anything useful with Yelp. :-( With your patch, we can. Mind you, there are still issues I'm trying to sort out and some ugly hackery taking place, but it sure beats no access. :-) So IMHO, yes it's worth fixing. :-)
I'm still trying to wrap my head around the various strange things we're seeing. As part of that.... from yelp-html.cpp: ---------------------------------- /* Fire "children_changed::add" event to refresh "UI-Grab" window of GOK, * this event is not fired when using gtk_moz_embed_xxx_stream, * see Mozilla bug #293670. Done in a timeout to allow mozilla to * actually draw to the screen */ static gboolean timeout_update_gok (YelpHtml *html) { g_signal_emit_by_name (gtk_widget_get_accessible (GTK_WIDGET (html)), "children_changed::add", -1, NULL, NULL); html->priv->timeout = 0; return FALSE; } ---------------------------------- As a result of the above, here's what gets emitted (after about a 2 second delay!) each time a new page is viewed by pressing Return on a link: <RETURN> (delay) object:children-changed:add(1, 0, [document frame | Sound & Video]) source: [application | ] application: [application | yelp] object:children-changed:add(-1, 0, None) source: [document frame | Sound & Video] application: [application | yelp] <RETURN> (delay) object:children-changed:add(2, 0, [document frame | Development]) source: [application | ] application: [application | yelp] object:children-changed:add(-1, 0, None) source: [document frame | Development] application: [application | yelp] <RETURN> (delay) object:children-changed:add(3, 0, [document frame | Games]) source: [application | ] application: [application | yelp] object:children-changed:add(-1, 0, None) source: [document frame | Games] application: [application | yelp] i.e. an additional event seems to be getting triggered/emitted, and it is causing the addition of more accessible children than we should have: If at this point you chose the document frame, get the reported parent (the nameless application accessible), and look at each of its children, you'll find we have instances of an accessible document frame named "Games". So I'm wondering: 1. Should a different event be emitted (one which doesn't result in bogus duplicate children)? 2. Should a corresponding object:children-changed:remove event also be emitted? 3. Should the underlying Mozilla bug be fixed instead? :-) It seems to be rather old and forgotten about....
Another issue -- one that I think will be hard for us to work around in Orca is this: Given an accessible object, calling getApplication() should return an object of the type Accessibility.Application. In yelp (even with the patch), it seems to return an object of the type Accessibility.Accessible (of ROLE_APPLICATION). Querying the Application Interface on that accessible largely fails with a NotImplementedError. :-( Ginn, any ideas on this one?
For comment #17, the fix is fragile, any call to relationset or refchild of the fake Gecko root acc will break it. For comment #18, object:children-changed:add(3, 0, [document frame | Games]) source: [application | ] application: [application | yelp] object:children-changed:add(-1, 0, None) source: [document frame | Games] application: [application | yelp] We've two problems here. 1) The number of children of nameless application keeps growing. That's a bug of Gecko. It's broken when it's embedded. 2) children-changed:add(-1) event This is an old workaround of GOK. The underlying bug of Gecko is not fixed, but the a11y code of Gecko was changed, so it gets around. We don't need this hack now. For comment #19, I can't reproduce it. I did, print acc.getApplication().queryApplication(), it seemed it's fine.
(In reply to comment #20) > This is an old workaround of GOK. > The underlying bug of Gecko is not fixed, but the a11y code of Gecko was > changed, so it gets around. > We don't need this hack now. I think the hack is still doing something though. If I remove it, caret-moved events from tabbing seem to go away after a new page is loaded. Are you seeing that as well? > For comment #19, > I can't reproduce it. > I did, print acc.getApplication().queryApplication(), it seemed it's fine. Trying it now in Accerciser, it seems fine for me as well. But when done in Orca I'm seeing failures. Maybe we're holding on to a cached object. I'll investigate that further.
(In reply to comment #21) > Trying it now in Accerciser, it seems fine for me as well. But when done in > Orca I'm seeing failures. Maybe we're holding on to a cached object. I'll > investigate that further. I dug into this a bit today. It's a slippery one. If I run the yelp_bug.py application by itself, things look good whether I run yelp before or after I run yelp_bug.py: event.host_application: application [application | yelp] event.source.getApplication(): [application | yelp] Things also look good if I run Orca after yelp is already running (not your typical use case). But... If I run Orca first, then yelp, something bizarre happens in yelp - we don't get the application name any longer. If I then run yelp_bug.py in this particular scenario, it's also getting bad information: event.host_application: application [application | ] event.source.getApplication(): [application | ] So...this leads me to believe that there is some event that Orca is getting and some sequence of calls Orca is making that is causing yelp/Gecko to get confused or bypass something that sets up the a11y hierarchy for us. I'll dig deeper to see what events or method calls might be causing this behavior.
Created attachment 118138 [details] New test app to force the no-name issue This test app builds upon the previous app --- it basically does some hierarchical operations on the object of a window:activate event. As a result of these operations, the name of the application ends up disappearing. So, you'll see output like this: caret moved: paragraph event.host_application: application [application | ] event.source.getApplication(): [application | ] Instead of this: caret moved: paragraph event.host_application: application [application | ] event.source.getApplication(): [application | ] We might be able to workaround this in Orca. Ginn, if you have any ideas, though, it would be great to try to solve this closer to the source of the problem.
Wait...sorry about that. I'm tired and mis-analyzed. Let me try again...
OK, let me try again: This test app builds upon the previous app --- it basically does some hierarchical operations on the object of a window:activate event. As a result of these operations, the name of the application ends up disappearing. So, you'll see output like this (no application name): caret moved: paragraph event.host_application: application [application | ] event.source.getApplication(): [application | ] Instead of this (application name of 'yelp'): caret moved: paragraph event.host_application: application [application | yelp] event.source.getApplication(): [application | yelp] We might be able to workaround this in Orca. Ginn, if you have any ideas, though, it would be great to try to solve this closer to the source of the problem.
I believe Mozilla is the best place for the fix. But I don't a good idea of how to fix it cleanly so far.
(In reply to comment #26) > I believe Mozilla is the best place for the fix. > But I don't a good idea of how to fix it cleanly so far. Fair enough. It's definitely a hard problem. The current patch (http://bugzilla.gnome.org/attachment.cgi?id=116420), however, does get us very very far. In fact, it permits us to provide access to help content again, which is something we haven't been able to do on GNOME for a long time. Yelp maintainers -- we've tested with the patch on this bug quite a bit. Can you please check it in so we can get it for GNOME 2.24? Thanks everyone for your hard work. :-)
(In reply to comment #27) > Yelp maintainers -- we've tested with the patch on this bug quite a bit. Can > you please check it in so we can get it for GNOME 2.24? Sorry to be a noodge here, but can we please get this patch in for GNOME 2.24?
Couple of quick questions: 1. Is the patch gecko 1.9 specific? (If so, I'll put it behind the GECKO_1_9 define in yelp) or is it applicable to 1.8 as well? 2. The timeout workaround isn't required for 1.9, right? If so I'll mask it behind a !GECKO_1_9 define. I've committed based on these assumptions. If these are wrong, let me know and I'll fix (leave a comment here or open a new bug). Otherwise, closing. Many thanks to everyone for investigating. 2008-09-11 Don Scorgie <dscorgie@svn.gnome.org> * src/yelp-html.cpp: Add workaround for Gecko 1.9 accessibility issue bug #545162 - Ginn Chen
(In reply to comment #29) > I've committed based on these assumptions. If these are wrong, let me know and > I'll fix (leave a comment here or open a new bug). Otherwise, closing. Thanks Don! Ginn is the best person to answer your questions about the patch. Us Orcians are pleased (and our users will be pleased) to have this checked in, though. Thanks so much! It will be a big milestone to have help accessible for GNOME 2.24.
Don, I have to echo what Will said, on all fronts: We *really* appreciate it, it is indeed a big milestone for GNOME. And Ginn is definitely the "go to guy" for all things Gecko. :-) (In reply to comment #29) > 2. The timeout workaround isn't required for 1.9, right? If so I'll mask it > behind a !GECKO_1_9 define. Wrong I'm afraid. :-( As I had suggested in comment #21: > (In reply to comment #20) > > This is an old workaround of GOK. > > The underlying bug of Gecko is not fixed, but the a11y code of Gecko was > > changed, so it gets around. > > We don't need this hack now. > > I think the hack is still doing something though. If I remove it, caret-moved > events from tabbing seem to go away after a new page is loaded. Are you seeing > that as well? I was able to verify that the problem exists by doing the following: 1. From within gnome-terminal, press F1. 2. Tab and arrow around in the resulting help content 3. Press Enter on the Introduction link 4. Tab and arrow around in the resulting help content In step 2, everything works as expected which suggests that Ginn's patch is doing its thing (because nothing worked as expected before :-) ). However step 4 fails miserably. We do not get a single caret-moved event (or for that matter focus and/or state-changed:focused events for links). I dunno what exactly that hack is doing to cause us to get the appropriate events, but it appears that it is still necessary. Reopening with a request that the change to the timeout hack be undone and a new commit be made for 2.24. Thanks!!
Timeout workaround is now done in all cases. Thanks. 2008-09-11 Don Scorgie <dscorgie@svn.gnome.org> * src/yelp-html.cpp: Reinstate previous timeout workaround as it's still required
Confirmed. Awesome! Thank you so much Don!
Joanie, I think the trick is in the timeout, it will do gtk_widget_get_accessible. I've another patch, it should work. And it tries it best to restore the parent accessible. As I said before, it's still possible that Gecko will set it back. I also moved the app root hacking from close to realize. It's a better place, in case something called gtk_widget_get_accessible() to html container before html is closed. Joanie, please do some test against it. Thanks!
Created attachment 118583 [details] [review] patch 2
Ginn, I just gave it a *brief* bit of testing and it seems a lot peppier. It also seems that we're getting more of the events we expect and thus should be able further improve things on our end. Yea! That said, I'm still working on my DayJob(tm) atm and want to test it more thoroughly. I'll do that this evening. Don, when were you planning on doing the Yelp tarball? Thanks Ginn!!
Reopening. Hard code freeze is on Monday (15th Sept.). Tarballs are then due a week later. So, ideally I'd like to commit the patch by Monday.
Ginn, one thing I'm noticing is that each time *any* link is followed, we're getting all of the events associated with a new page being loaded. If the link being followed is for content on the same page, the document really isn't loading (is it?). If possible, it would be helpful to just get the loading-related events when a new page is about to be displayed. If not possible, I will look for a way around it. Thanks!
I think it's not possible. No matter what link it is, the whole html is sent to Gecko. So Gecko would not know.
Ginn and Don: I think Ginn's patch is a great improvement and should be committed. I've also got a corresponding Orca patch ready to go which seems to work quite nicely. That said, I'd be more comfortable if Will were to chime in with his opinion. After all, we're so very close to code freeze, and what we have currently does work well enough. Will, what say you? :-)
(In reply to comment #40) > Ginn and Don: I think Ginn's patch is a great improvement and should be > committed. I've also got a corresponding Orca patch ready to go which seems to > work quite nicely. That said, I'd be more comfortable if Will were to chime in > with his opinion. After all, we're so very close to code freeze, and what we > have currently does work well enough. > > Will, what say you? :-) I say go for it. :-)
Patch committed. Huge thanks. 2008-09-15 Don Scorgie <dscorgie@svn.gnome.org> * src/yelp-html.cpp: Improved a11y under gecko 1.9 - Yet more work on bug #545162 from Ginn Chen