GNOME Bugzilla – Bug 356702
GOK failed to grab the HTML widgets.
Last modified: 2007-07-09 15:01:21 UTC
Please describe the problem: GOK could not grab the widgets in HTML page in firefox3. Steps to reproduce: 1. Invoke firefox 3 and open the attached html test page. 2. Launch GOK. 3. Select UI Grab in GOK main. Actual results: The widgets in this page can not be grabbed in GOK. Expected results: The GOK could grab these widgets contained in HTML page. Does this happen every time? Yes. Other information: This bug can be reproduced with GOK1.2.0 and Mozilla/5.0 (X11; U; SunOS i86pc; en-US; rv:1.9a1) Gecko/20060919 Minefield/3.0a1 on vermillion_49a/snv_48.
Created attachment 73027 [details] test page
Tim where are you grabbing firefox 3 from?
Hi David, you can download the firefox 3 in mozilla community with the link: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/contrib/ . This will bring you to the firefox trunk build for solaris. If you want to find other distributions of firefox trunk build, please go to http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ .
David, I guess the cause is GOK doesn't dig into acc children if the parent implements hypertext interface. In Firefox 3, we made some significant changes to Gecko a11y.
Tim, Ginn, I'm going to be able to spend some time next week on firefox. In the meantime, are you guys able to build GOK locally? If so, and if you dare, you could bravely take a look at some gok code and see if you can fix this. Perhaps in gok_keyboard_branch_byKey (in gok-keyboard.c)... look for case KEYTYPE_BRANCHHYPERTEXT... Sorry I can't be more help today.
Tim, Ginn, I'm going to be able to spend some time next week on firefox. In the meantime, are you guys able to build GOK locally? If so, and if you dare, you could bravely take a look at some gok code and see if you can fix this. Places to look: gok-keyboard.c - "gok_keyboard_follow_link" gok-spy.c - "gok_spy_append_node" - look for comment: /* Hypertext objects are treated differently; one node per link */ The way gok deals with hyperlinks was kludgey (by necessity) and needs refactoring. Sorry I can't be more help today.
Tim, Ginn, can you reconfirm this bug?
I can confirm this bug with GOK Head. I got a lightblue pressed button with "FFFC" on it for HTML container/frame. Your statement in #c6 is correct. But I don't have time to solve it.
OK Thanks Ginn. I'll try to look at this Friday (my regular GOK day).
Created attachment 89187 [details] screenshot (at-poke and minefield) OK confirming here too. I'm not sure what this FF,FC square character is.
(In reply to comment #10) > Created an attachment (id=89187) [edit] > screenshot (at-poke and minefield) > > OK confirming here too. I'm not sure what this FF,FC square character is. > FFFC would be the wonderful unicode EMBEDDED_OBJECT_CHARACTER, which is used by Gecko to indicate that the portion of text for the current object is to come from the child object. For each EMBEDDED_OBJECT_CHARACTER in the parent's text, there is guaranteed to be a child, and the order of the children matches the order of the EMBEDDED_OBJECT_CHARACTERs in the parent's text.
Created attachment 90955 [details] [review] fixes the yucky fffc and restores functionality that gok had for simple cases. I'm setting the initial status of this patch to needs-work. I'm posting it for two reasons: 1. I might be barking mad, anyone who cares to peek at it... let me know if the direction here is wonky. I need to, and haven't yet, taken a look at the gecko a11y utf-8 embed code; I need to find out what is expected of AT that is walking the tree. 2. I'm not happy with it, but I still want to keep it somewhere safe.
Let's zoom out to the big picture for a second. What does the best OSK for HTML/DHTML look and act like? IMO, it would be something akin to enhancement bug #117583. In the meantime, here are some options: 1. Create a tighter compose keyboard which has greater weighting set on HTML navigation and Ix keys: tab, shift-tab, pgup, pgdown, arrows, ctrl-tab, ctrl-w, ctrl-pgup, ctrl-pgdwn. 2. Walk the DOM using AccessibleHypertext, AccessibleText and grab AccessibleHyperlink objects, and build a keyboard out of them. We do #2, in such a way that required some FF hacks which have been removed... thus this bug report. #2 is neat, it makes for good demonstrations of FF interactivity, and more importantly in some cases is probably a good user experience. In some cases however, the links, represented by gok keys are confusing or ambiguous as they are out of context. A solution like #1 might be a good low-maintenance solution until we have something like 117583. I'm not sure.
option #1 sounds good. can we have both? e.g. It's convenient if we have pgup/pgdown, since Fx doesn't make scroll bar accessible yet. Even Fx makes scroll bar accessible, I think pgup/pgdown is easier to use.
Let's aim for both (at least eventually). I'm giving #2 a good try (actually a variation on #2) and am making good progress. I think it will work for us at least as well as it used to.
David, I agree with your thrust toward #2. UI Grab typically shows more than just hyperlinks - grabbing the scroll bars as well for rapid navigation. GOK should work with FF at least as well as it does with yelp (which after all now uses the same gecko engine...). Proposal #1 is interesting... I think it is worthy of some user testing. It moves us away from GOK as a totally generic keyboard, and into app-specific behavior. Nothing wrong with that, but I imagine a wholesale refactor may be in order in GOK to support that well (e.g. a re-write in Python, more along Orca lines).
Created attachment 91142 [details] [review] potential fix; needs more testing. Peter, Ginn, thanks for your comments. I agree with both of you :) Anyways, here's my patch so far. It seems to capture everything (not just links). I am considering changing the "look" of the keys for html based GUI but not sure. Also I would like to hunt down and remove more hackery that is no longer needed. This could probably go in but I would like clean it up a bit tomorrow am.
My tests have uncovered a problem with the updating of the keyboard to reflect a newly visible page tab scroll panel that only seems to occur in FF. FF (and xul generally I supose) has a page tab list which contains the page tabs. The containers (scroll panes) are _siblings_ (no handy hierarchical relationship) to the page tabs. Also, something may be a bit wonky with gecko event order. I hope to debug this during my current 2 hour meeting (shhh).
Created attachment 91289 [details] [review] best user experience so far... OK. After investigating several approaches I'm getting very close. One thing I'm not catching is when a link in a browser tab is activated and a new page is loaded. GOK will invalidate the keys it has representing the previous page but won't re-walk the tree. I'll need to look for which events will signal a new web page load. Note Tim's test page has some checkboxes that end up with no accessible name... so GOK won't display them; although we can see them by running 'gok --debugnameless' I'm thinking I'll put this patch in with little change late Friday.
Fixed via rev: 2417 I revved the version to 2.3.0 for gnome 2.19 and ff3+ Created a gnome-2-18 branch for gnome 2.18 and <ff3 Ginn, Peter, Tim, thanks for your help.