GNOME Bugzilla – Bug 423427
[Pending] Need to add form field structural navigation in Firefox
Last modified: 2008-07-22 19:27:18 UTC
The Orca specification for Firefox says: > The letters "f" and "Shift+f" will be used to move between form fields. It also says: > links and form controls will not be considered large objects. This is not yet implemented.
Created attachment 86084 [details] [review] first pass at this Mike: Could you please give this a shot? Also, I went ahead and changed the order of how combo boxes are spoken as you specified: Now it is label (if any) text displayed combo box, whereas before it was label (if any) combo box text displayed. I also believe I have lists being spoken the way you wanted. The next thing on the form field front is inferring labels from HTML content (bug #407941) Will: Regarding the issue we had discussed where grabFocus was returning True, and Orca knew where we were, but Firefox didn't (thus pressing Tab after using F/Shift F gave some undesirable results): Seems you cannot grabFocus on just any component and expect Firefox to attend to the change. :-) This patch passes the Tab-after-F'ing test. (That doesn't sound quite right, but you know what I mean) ;-)
Created attachment 86085 [details] [review] changed the keybinding from F/Shift+F to Orca+Tab/Orca+Shift+Tab You follow a spec and what do you get? A phone call from the spec writer saying "on second thought...." <huge grin> Mike suggested that F/Shift+F would cause confusion for the user because they would get stuck in entries typing Fs. He proposed Orca+Tab/Orca+Shift+Tab instead. Done. Mike please let me know what you think about this version.
Mike pointed out a problem he found: When you're commenting on a bug, Insert+Tab moves you from the "Add me to CC list" checkbox to the text entry for your comment to the "points" link of the person who filed the bug. That's no good. :-( Changing the status of the patch to "needs-work". I'll investigate more tomorrow.
Ya know.... It's not that it's jumping from the entry to "points," per se: It's jumping to the first focusable item that falls after the location where you were when you initially pressed Insert+Tab. That makes me think that this is another version of the problem I mentioned in the comment associated with the initial patch: Currently I'm only doing special grabFocus handling of lists and combo boxes. Methinks that might need extending to other form fields -- checkboxes and entries to name a couple. :-)
> Will: Regarding the issue we had discussed where grabFocus was returning True, > and Orca knew where we were, but Firefox didn't (thus pressing Tab after using > F/Shift F gave some undesirable results): Seems you cannot grabFocus on just > any component and expect Firefox to attend to the change. :-) Do we still run into the problem where FF tells us a component is FOCUSABLE, returns True when we do a grabFocus on it, but the component doesn't end up with focus?
> Do we still run into the problem where FF tells us a component is FOCUSABLE, > returns True when we do a grabFocus on it, but the component doesn't end up > with focus? Sorta.... What *seems* to be the case is that the component often does end up with focus (e.g. we can check the check box we're on). What we're getting when we Tab to an object that we don't get when we move to that object some other way and grabFocus is a caret-moved event. Firefox thinks the caret is still off wherever it was before. Finding some more hacks to solve this problem on our end.... Stay tuned.
Created attachment 86121 [details] [review] a bit hacky, but.... Okay in this version: 1. useStructuralNavigationModel() checks to see if MODIFIER_ORCA is being pressed. We weren't doing that before because until now we didn't have structural navigation keys involving MODIFIER_ORCA. Prior to this addition, our keystroke was being processed as a regular old Tab. This change should stay in. 2. Added a hack for radio buttons and checkboxes. The hack logic is this: If we're on a radio button or checkbox in a form, there's bound to be some text nearby and on the same line identifying what the blessed thing is. Therefore, if we're heading forward, look forward for the next instance of text; if we're heading backward, look backward to the previous instance of text. Place the caret there. That should wake Firefox up. THEN place the caret on the form field so that we can interact with it. This ideally will place us in between our current radio button/checkbox and the next form field in order. This hack will presumably fail in the presence of a bunch of checkboxes or radio buttons with no text nearby. I guess it's a gamble. Do we want to take it? 3. Added a hack for entries and password_text. Perform an obj.action.doAction(indexOfActivateAction). Works on single-line entries but not on multi-line ones. Go figure.... At first I thought we could uber-hack multi-line ones with an eventsynthesizer.clickObject. And it generally worked. But then, somehow, in my attempts to move to an entry, I clicked on the embedded object to switch desktops. That is bad. Very bad. So we're going to have to punt and file a bug against Firefox. Note: I still haven't figured out a hack for push buttons. Unlike radio buttons and checkboxes, we cannot count on nearby text on the same line. Who knows where we will land? Thus hack #2 is out. I'm open to suggestions. Haven't tried spin buttons at all yet....
Hey Joanie: Wow. This is a lot of work. Thanks for doing it. Do you have a test page for this? In addition, aside from enumerating all the possible form control like things, is there a way to infer something is a form control (e.g., it's a descendant of some object of some role)?
> Do you have a test page for this? I've been using four primarily: 1. simpleform.html from our test case pages: http://svn.gnome.org/svn/orca/trunk/test/html/simpleform.htm 2. Bugzilla's enter new bug form 3. The form at the bottom of existing bugs (where you can change the status, close it out, etc.) 4. The three search forms at the NLS. The direct link is ugly, but go to: http://www.loc.gov/nls/index.html and follow the link to search the catalog. There are three different forms there. Then spot-checking elsewhere. > In addition, aside from enumerating all the possible form control > like things, is there a way to infer something is a form control > (e.g., it's a descendant of some object of some role)? Oh, I dunno, you mean like ... hmmm.... ROLE_FORM? ;-) ;-) ;-) Are you thinking that if it's a descendant of ROLE_FORM, and it's focusable (but not a link), goNext{Previous}FormField should move you there so that we don't skip over some control we didn't include? Or is your question more related to the control-specific hacks which (I believe) are necessary? BTW, the testing makes it really, really clear that *someone* needs to be inferring form fields....
> I've been using four primarily: Cool. Thanks! Gets us on the same {web} page. > > In addition, aside from enumerating all the possible form control > > like things, is there a way to infer something is a form control > > (e.g., it's a descendant of some object of some role)? > > Oh, I dunno, you mean like ... hmmm.... ROLE_FORM? ;-) ;-) ;-) > > Are you thinking that if it's a descendant of ROLE_FORM, and it's focusable > (but not a link), goNext{Previous}FormField should move you there so that we > don't skip over some control we didn't include? Yeah...something like that. :-) > Or is your question more > related to the control-specific hacks which (I believe) are necessary? Yeah, we're going to have to hack. :-( It looks like you've identified the hacks well via documentation, though. Mike, have you tried this out?
On the site https://www.bankofamerica.com/index.jsp I frequently hear nothing when trying to use this feature. I hear the edit fields but often stop on elements that I don't know what they are. It seems to work well on most sites though.
Confirmed on the BOA site. Let me see what I can figure out. Thanks for testing it!
The entry issue is worked out: The activate action which is part of my entry hack has the nice side effect of triggering a focus: event. The focus: event causes Orca to speak things like entry labels and any existing entry text, update the braille, and so on (i.e. to do what we expect). Subsequently calling setCaretPosition() and speakContents() was both redundant and interrupting the speaking of labels. Returning upon success of the activate action makes things much better. Thanks for testing! Rather than post a patch now and then implement Will's suggestion from comment 8, I think I'll wait until after I implement Will's suggestion. Stay tuned!
Created attachment 86721 [details] [review] fixed the entry issue; implemented Will's suggestion This version should solve the issue with respect to entries identified by Mike. For the sake of documentation/explanation: What you should hear when using form field structural navigation to navigate to a form field is the same thing you hear when you navigate to the same form field using Tab/Shift Tab. In other words, if you hear the entry label when you Tab to the entry, you should expect to hear it when you Orca_Modifier+Tab to it. If there's a difference, it's a bug -- as was the case with the Bank of America fields. (Thanks again Mike!) If you don't hear a label either way, then it may be a bug elsewhere or -- more likely -- it's because the form field is not labeled properly and someone needs to implement some label-guess heuristics. It's on the Firefox plate (https://bugzilla.mozilla.org/show_bug.cgi?id=376481) and the Orca plate (http://bugzilla.gnome.org/show_bug.cgi?id=407941). I plan to start looking at it from our perspective quite soon (like this weekend) as we need something implemented. Will: I implemented your suggestion of not enumerating all the possible form control like things. In the process of doing so, I turned getContainingLink() into getContainingRole() so that we can use it for other things (e.g. to get the containing Form, to get the parent whatever of a nested whatever, and -- perhaps -- to get the containing heading of a link as part of the fix for bug #414657, etc.) Thoughts?
The latest patch nicely fixes the previus problem I reported. I'm still noticing that when I use this feature on this bug page that the tab order doesn't properly update when moving to edit fields. For example, if I use this feature to move to the edit field and then tab I will still be tabbing from the top of the page.
If you mean the multi-line entry for comments, that is a known issue. The activate action fails on multi-line entries and, as a result, we cannot update our position in the tab order. I will be filing a bug against Firefox for that issue soon.
You know.... I just tried the following: 1. Load this page 2. Press Orca_Modifier+Tab. I moved to the Add me to CC list 3. Press Orca_Modifier+Tab. I moved to the multi-line entry in which one adds comments. 4. Press Tab. I landed on the "need_stacktrace" link (which is what I would want and expect) So.... Can you give me more specific steps to follow so that I can be sure that what you're seeing is not a bug in my patch? Thanks!!
With respect to the failure of the activate action on multi-line entries, I have filed this bug against Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=378408. We're tracking it with bug #432432.
> Will: I implemented your suggestion of not enumerating all the possible form > control like things. In the process of doing so, I turned getContainingLink() > into getContainingRole() so that we can use it for other things (e.g. to get > the containing Form, to get the parent whatever of a nested whatever, and -- > perhaps -- to get the containing heading of a link as part of the fix for bug > #414657, etc.) Looks good. Thanks!
I am nolonger able to reproduce the above problem. Perhaps you should check this in so more people can start trying it. thanks much
> Perhaps you should check this in so more people can start trying it. Done. I also sent out a note to the list explaining the new feature and asking for feedback. Thanks!
Created attachment 86918 [details] [review] be sure the form field has state of SENSITIVE and is not role UNKNOWN In the determination of what constitutes a form field we want to navigate to, be sure it does not have a role of UNKNOWN. Also be sure it has a state of SENSITIVE. (Test case: http://www.sj.se/sales/searchLocations.do Thanks Krister!!)
Figured I should summarize where things are with this for documentation: While there may be some bugs here and there around where we are landing, the feature seems over all to be working as expected with one big ol' exception: If the user is using Desktop layout, is in an entry, wants to move to the previous form field and fails to get Tab held down more or less simultaneously with Insert and Shift, zeros get typed into the entry. :-( Hermann identified that a work-around was to use the other Insert key. Mike wants to find a way to make this key combination work if at all possible rather than looking for a different keystroke. Do we need to use xmodmap to map KP_Insert to Insert? Are there better (easier, non-mapping) approaches? Lorenzo raised the issue that "fixing" our feature means "breaking" another feature, namely that holding down Shift means that you can ten-key in numbers without having to use NumLock. This didn't seem to bother anyone on the list. Rich suggested we ask Will when he gets back from vacation. :-)
The fix for bug 434948 seems to resolve the 0's problem. So...maybe we can close this bug out and file specific bugs for specific problems related to this one as we find them?
The one remaining issue is the fact that we do not always update our position correctly. As best as I can tell, this is due to two or three Firefox bugs: 1. Combo box contents up and became list items a couple of weeks ago. This means we think we're in a numbered or bulleted list and all bets are off. 2. grabFocus() doesn't, uh, grab focus.... :-) 3. the activate action fails on multi-line entries (we wouldn't care if it weren't for bug #2) Aaron seems to be hard at work with proposed fixes for all three. Once they are checked in, I will verify that they are working and adjust go{Next, Previous}FormField() accordingly. I will do this whether or not this bug remains open. :-) So as far as I'm concerned it can be closed. Any objections Mike?
You know what, Aaron's patches -- at least some of them -- seem to have made it into this afternoon's build. Since I will need some place to attach our resulting patches, and since this is still open, how 'bout we leave it open a wee bit longer and I'll attach those patches here? :-)
It looks like the grabFocus() and activate action patches didn't make it into today's build -- that's my hope anyway since that functionality is still not in place. :-( Hopefully it will be in tomorrow's....
Alrighty, it's adjustment time! I'm going to do these as individual patches because we have several new issues that need sorting out. Here's issue and adjustment #1: We have not been able to provide decent (let alone compelling) support for the Find toolbar due to a complete and utter lack of events as the caret moves within the document frame during the search. See bug #407663 for more info. Now we get caret-moved events for non-focused items which will allow us to remedy that problem. (Yea!) But we also now get caret-moved events for other things which we need to start actively ignoring. In particular when a form field gains focus, some other item (seems to be the parent form or parent table cell) issues a caret-moved event. We were updating the locusOfFocus unconditionally and then chasing after these events. THIS is why we weren't able to select items in combo boxes as of today's Firefox build. It is also why, as Mike discovered, you could no longer type in a multi-line entry. Mike, could you please try this patch and see if your form field woes go away (and that nothing else odd happens <grin>)? Thanks much!! 1 down.... (I hope) :-)
Created attachment 87844 [details] [review] aforemention patch which I forgot to attach :-) D'oh! Too much staring at Firefox. Mike could you please try *this* patch? Thanks!
Created attachment 87847 [details] [review] adjustment patch #2 Okay, *this* patch removes much of the ugly -- and not always successful -- hacking we had to do as a result of grabFocus() failing to grabFocus(). I removed the hacks for checkboxes, radio buttons, and entries/password text. I have left in the hacks for the combo boxes and lists for now: In part because I want to be sure that the previously attached patch is sound before basing a new patch on it; in part because it's still not 100% clear to me what the children of lists will be when all is said and done. :-) The goal when testing this patch is to be sure that we are properly updating our location when using form field structural navigation (not counting combo boxes and lists). Mike could you please give this one a spin as well? Thanks!
This seems to be working much better. The only thing I've noticed is that before hearing that the control where I am currently typing was announced I heard the title of the page. This happened when I pressed insert+tab to get to the edit control. Because I get this behavior some times anyway I don't think it is related to this patch though.
> This seems to be working much better. Thanks to which patch? Or both? I'm guessing from the time stamp on your comment that you are just talking about the patch in comment 29 rather than the one in comment 30. In which case we still need to verify that we are updating our position correcting when using form field structural navigation. Also please note that those patches are independent of one another. Form field structural navigation was impacted on several different fronts by all of Aaron's patches from the past couple of days -- mostly it's a positive impact, mind you, but one that requires us to adjust how we do things. Rather than open new bugs for each adjustment, I'm just doing separate patches to this one since it was still open. Sorry for any confusion! Will, can you think of anything I'm overlooking in the patch to comment 29? I'm a little leery about saying, "nah, don't update the locusOfFocus this time" -- especially after you said, "always do so." ;-)
> Will, can you think of anything I'm overlooking in the patch to comment 29? > I'm a little leery about saying, "nah, don't update the locusOfFocus this time" > -- especially after you said, "always do so." ;-) I think it looks fine. I think the original assumption in the code was that the thing in which the caret moved was always the locus of focus. The main problem I think this was addressing was that we wouldn't always get focus events for the thing with focus, so we would just force it when we saw a caret moved event. Now that the caret can move in things that don't have focus, this assumption is no longer correct. So...what you're doing is probably fine.
Created attachment 87922 [details] [review] unified patch (the last if we're lucky) Okay, I believe I have taken care of all the form-field-navigation changes that needed to be made as a result of all of the changes Aaron made. With this patch we should see: 1. Accurate updating of locations when using structural navigation 2. Proper handling of lists and combo boxes 3. The ability to enter text into entries Basically, I think this does it -- assuming it all works like I think it does. :-) Mike please give this a go and let me know what you think. Will, thanks for the feedback. I came up with a case I didn't think of: If focus is in a form field and the user does a mouse-click to move the caret to non-focusable text within the document frame. I KNEW there had to be something I was overlooking. :-) :-) This patch handles that case.
Hey Joanie, so far I've tested this on: slashdot, bank of america, craigslist, bugzilla, audible, and google. It seems to be working really well. I'd say commit it. Thanks a lot for doing this. It really is working much much better.
(In reply to comment #35) > bugzilla, audible, and google. It seems to be working really well. I'd say > commit it. Three words: Woo, hoo, done. :-) Thanks for testing it! (In reply to comment #24) > The fix for bug 434948 seems to resolve the 0's problem. So...maybe we can > close this bug out and file specific bugs for specific problems related to this > one as we find them? *Now* I feel comfortable closing this one out. :-)
Closing! Thanks everyone!