GNOME Bugzilla – Bug 486908
Selection and navigation in multiselectable items are not properly handled.
Last modified: 2008-07-22 19:33:30 UTC
Steps to reproduce: 1) Run the Icon View -> Icon View Basics demo of gtk-demo 2) Arrow down to the first icon. Press Shift+Right to select two icons. Orca should present the new selection state to the user if the number of selected items exceeds one.
See also the problematic test/keystrokes/gtk-demo/role_icon.py. This test is problematic in that its display varies depending upon the platform you are using, but it at least demonstrates the problem.
Created attachment 99028 [details] Orca debug output generated whilst running the example. Mike/Will, what do we want brailled/spoken here? As it currently stand, when you shift-right to extend the selection to a new icon, it'll just braille/speak the name of the last icon. But there will be more than one icon selected. Should we state how many icon are selected, or just the selected-ness state of the icon that has focus?
(In reply to comment #2) > Mike/Will, what do we want brailled/spoken here? Definitely a question for our UI guy. I'll defer to him. :-)
We should only be speaking the last item selected. WhereAmI will allow the user to get more detailed information on the number of items selected.
Created attachment 99055 [details] [review] Revision #1. Patch to hopefully fix the problem. If this is what is wanted, I'll also adjust .../test/keystrokes/gtk-demo/role_icon.py
Saw the "testing required". :-) More questions for Mike: Test/Question 1: 1. Start with Will's reported test case (i.e. the first two icons in the first row are selected). 2. Press Shift+Down thus simultaneously selecting two icons in the second row. What is now spoken is the last icon selected (lib32), as was specified. The fact that lib was also selected at the very same time is not indicated in any way. This *could* be looked at as the same question Rich posed in comment #2, but I'm not convinced that it is. :-) In the original test case, only one icon was being selected. Perhaps a more interesting example is to start from the "bin" icon and press Shift+End. This turned out to be one of those "I'll be darned" moments. What I expected to occur was that the first row would be selected (i.e. bin through initrd). Failing that, my second guess would have been that all icons would be selected. Listening to what Orca announced (vmlinuz selected), my non-visual conclusion would be that my second guess was correct. After all, I was on the first icon; Orca just announced the last icon was selected. I must've gotten them all, right? Heh. Nope. Turns out I selected the first three icons in each row and failed to select the last four icons rows one through three. Should Orca have given me any indication to that effect (in the form of which icons were selected), or should the user selecting multiple icons at once know that he/she should use whereAmI to confirm the selection? Question 2: Orca announces when text has become unselected. We are not announcing when icons become unselected. Should we be, or are icons an entirely different case?
Adding "[mike]" to the Summary line so that he can see that Joanie has a load more questions.
> > 1. Start with Will's reported test case (i.e. the first two icons in > the first row are selected). > 2. Press Shift+Down thus simultaneously selecting two icons in the > second row. > > What is now spoken is the last icon selected (lib32), as was specified. The > fact that lib was also selected at the very same time is not indicated in any > way. This *could* be looked at as the same question Rich posed in comment #2, > but I'm not convinced that it is. :-) In the original test case, only one icon > was being selected. I still think we only want to speak the last icon selected which will be the one with focus I think. The user should use WhereAmI to determine if many icons are selected. > > Perhaps a more interesting example is to start from the "bin" icon and press > Shift+End. This turned out to be one of those "I'll be darned" moments. What > I expected to occur was that the first row would be selected (i.e. bin through > initrd). Failing that, my second guess would have been that all icons would be > selected. Listening to what Orca announced (vmlinuz selected), my non-visual > conclusion would be that my second guess was correct. After all, I was on the > first icon; Orca just announced the last icon was selected. I must've gotten > them all, right? Heh. Nope. Turns out I selected the first three icons in > each row and failed to select the last four icons rows one through three. > Should Orca have given me any indication to that effect (in the form of which > icons were selected), or should the user selecting multiple icons at once know > that he/she should use whereAmI to confirm the selection? Please see my above comment. The thing I would worry about if we were to speak all the selected icons is getting much more verbosity than the user would normally want. For example, consider the case where something like 30 icons get selected. Joanie's example here is exactly why I think most blind people avoid icon views whereever possible. > > Question 2: > > Orca announces when text has become unselected. We are not announcing when > icons become unselected. Should we be, or are icons an entirely different > case? I think we would want to speak unselected if the user is holding down CTRL while arrowing. For example: if focus is on a selected icon and the user presses space to unselect it. I think we should follow what we are planning for tree tables and speak unselected. > Given my comments here I wonder if we should extend our WhereAmI support for icon views to add a detailed description of the current state of selection. This could be a nice thing to add to the double press.
> I think we would want to speak unselected if the user is holding down CTRL > while arrowing. Isn't this going to be part of the fix for bug #486972 (as discussed in comments #15-21)? We seem to end up tagging a new kind of bug onto an existing one, and then these bugs stay open (and cause confusion) for ages. Could somebody, please summarize what (if anything) still needs to be done before this bug can be closed? I strongly suggest that we open up a new bug for the proposed selection changes in comments #15-21 of bug #486972 and then hopefully we can close bug #486972 (the original expanding/collapsing tree node problem is now fixed right?) and this one (we speak the last selected icon in the layered pane correctly, right?) Thanks.
> We seem to end up tagging a new kind of bug onto an > existing one, and then these bugs stay open (and > cause confusion) for ages. I agree with this: I generally like us to fix the bug specified in a bug report and open a new bug report to handle a new bug rather than just keep adding to a bug. Having said that, we should use our better judgment about the gray areas where a very closely related problem might just be better off solved via an existing bug. Unfortunately, it's not black and white and I don't think it ever will be. > Could somebody, please summarize what (if anything) > still needs to be done before this bug can be closed? I think a good thing to do would be to retitle this bug to "Selection and navigation in multiselectable items is not properly handled." In that, we have examples of things that need to be handled better: * Navigation via Ctrl+{Up,Down} to move the dotted rectangle * Ctrl+Space to select/unselect a node * {Up,Down} from an unselected node In general, I think we need to handle the various combinations of navigating, selecting, and unselecting via the arrow and page up/down keys. The two use cases we know of are the icon view and the tree view examples in gtk-demo. They seem very closely related to me and might be solvable with the same code. I think they should at least have a very similar user interaction model. If we discover that the code paths are remarkably different, however, or the user interaction model is remarkably different, then we should separate this in to two bugs.
Make that "Selection and navigation in multiselectable items *are* not properly handled."
Several comments related to the selection and navigation problems were added to bug #486972. Copying them here so that they are all in one place. ------------------- Comment #15 from Rich Burridge (orca developer, points: 21) 2007-11-06 21:48 UTC [reply] We discussed this bug in the Orca team meeting today. For the problem of not reporting anything when we initially Down arrow into the table, we are going to try out the following approach to see if it will fix the problem. (Note that if you Down arrow a second time, it will select that first entry.) If we are now selected but weren't previously, then we should speak/braille the expanded/collapsed state. Returning bug state to normal while this is investigated. ------------------ Comment #16 from Willie Walker (reporter, orca developer, points: 20) 2007-11-07 19:02 UTC [reply] Mike - I think we need some design input from you. This may turn into a separate bug, but the new situation is that the table in question has the following multiselect behavior as brought to our attention by Joanie: * A dotted rectangle around the row lets you know about the current cell of interest. There's only one dotted rectangle. Moving the dotted rectangle up and down without changing selection is managed via Ctrl+{Up,Down}. * The highlighting of a cell indicates if it is selected or not. There can be multiple discontiguous sets of cells selected. The toggling of selection for the current cell of interest is managed via Ctrl+Space. Selection is also implicitly done when just Up or Down are pressed. When just Up or Down are pressed, all selection is cleared and then the newly landed on thing (i.e., the thing that gets the dotted rectangle) becomes the sole thing selected. We need your user experience design on this. In particular, I think a general model what should be presented when: * Navigating the dotted rectangle to a cell -- I'm assuming the contents should be spoken following the "read table cell or entire row" setting. But, I'm assuming the indication of whether the thing is selected or not is also important and should be presented. What should the presentation be in speech and braille for this and under what conditions should the presentation be done (e.g., only when unselected?). * Expanding/collapsing a cell that's not selected/highlighted. I'm assuming something should be spoken here, and it should match the output we get when we expand/collapse the node when it is selected? * Changing the selection state of the cell. I'm assuming we should also present something here. What should the presentation be in speech and braille for this? For technical details... When the dotted rectangle moves, we seem to get object:active-descendant-changed events: OBJECT EVENT: object:active-descendant-changed detail=(126,0) app.name='gtk-demo' name=None role='tree table' state='ENABLED FOC USABLE FOCUSED SENSITIVE SHOWING VISIBLE MANAGES_DESCENDANTS' relations='' When the selection changes, we seem to get object:state-changed:selected events: OBJECT EVENT: object:state-changed:selected detail=(1,0) app.name='gtk-demo' name='Army Day' role='table cell' state='ENABL ED FOCUSABLE SELECTABLE SELECTED SENSITIVE SHOWING SINGLE_LINE TRANSIENT VISIBLE ' relations='NODE_CHILD_OF' NOTE: handling this might require some rethinking of logic in default.py. For example, in default.py:onSelectionChanged, there's this comment: # Avoid doing this with objects that manage their descendants # because they'll issue a descendant changed event. # if event.source.getState().contains(pyatspi.STATE_MANAGES_DESCENDANTS): return In the light of this multiselect behavior, the assumption made in the comment is false: selection can change for the current active descendant. ---------------- Comment #17 from Mike Pedersen (orca developer, points: 13) 2007-11-07 20:35 UTC [reply] > * Navigating the dotted rectangle to a cell -- I'm assuming the contents should > be spoken following the "read table cell or entire row" setting. But, I'm > assuming the indication of whether the thing is selected or not is also > important and should be presented. What should the presentation be in speech > and braille for this and under what conditions should the presentation be done > (e.g., only when unselected?). If the item is selected it should be underlined in braille honoring orca's existing braile marking setting. In speech "selected" should be the last thing spoken after the information which is currently presented about the cell or row. If an item is not selected nothing different should be spoken or brailed. > > * Expanding/collapsing a cell that's not selected/highlighted. I'm assuming > something should be spoken here, and it should match the output we get when we > expand/collapse the node when it is selected? How would one perform an action on something that wasn't selected? > > * Changing the selection state of the cell. I'm assuming we should also > present something here. What should the presentation be in speech and braille > for this? In braille the marking should be shown or taken away based on the new selection state. In speech "selected" or "unselected" should be spoken. With speech the cell/row should not be repeated when the selection changes. ----------------- Comment #18 from Willie Walker (reporter, orca developer, points: 20) 2007-11-07 22:35 UTC [reply] > > * Expanding/collapsing a cell that's not selected/highlighted. I'm assuming > > something should be spoken here, and it should match the output we get when we > > expand/collapse the node when it is selected? > How would one perform an action on something that wasn't selected? There are two things happening here: 1) There is a dotted rectangle around the row indicating it is the thing of interest. Any keyboard action typically operates on this object. 2) There is a separate notion of selection. When something is selected, it is highlighted. Yes, it is confusing. I believe the reason things are done this way is to allow keyboard users to create multiple discontiguous selections. For example, if I want to select rows 4 and 6 of a table, I would: 1) Arrow to row 4. This would select 4 as a side effect of an unmodified arrow operation (i.e., I didn't hold the control key down) doing both a select and a move. 2) Ctrl+arrow to row 6. This would leave row 4 highlighted and move the dotted rectangle to row 6. 3) Press Ctrl+Space. This would select row 6, also leaving row 4 selected. The dotted rectangle remains on row 6. -------------------- Comment #19 from Mike Pedersen (orca developer, points: 13) 2007-11-08 00:36 UTC [reply] > There are two things happening here: > > 1) There is a dotted rectangle around the row indicating it is the thing of > interest. Any keyboard action typically operates on this object. > > 2) There is a separate notion of selection. When something is selected, it is > highlighted. > > Yes, it is confusing. I believe the reason things are done this way is to > allow keyboard users to create multiple discontiguous selections. For example, > if I want to select rows 4 and 6 of a table, I would: > > 1) Arrow to row 4. This would select 4 as a side effect of an unmodified arrow > operation (i.e., I didn't hold the control key down) doing both a select and a > move. > > 2) Ctrl+arrow to row 6. This would leave row 4 highlighted and move the dotted > rectangle to row 6. > > 3) Press Ctrl+Space. This would select row 6, also leaving row 4 selected. > The dotted rectangle remains on row 6. > So if I am understanding this case correctly the only thing we should speak is the state change because we have already spoken the item with focus. Am I correct or did I miss something? --------------------- Comment #20 from Willie Walker (reporter, orca developer, points: 20) 2007-11-09 17:40 UTC [reply] > So if I am understanding this case correctly the only thing we should speak is > the state change because we have already spoken the item with focus. > Am I correct or did I miss something? I think this sounds about right. In the team meeting, I think we also agreed to leave this bug open as a means for tracking the rework of dealing with active-descendant and selection change events when working with multiselect objects like this. ----------------------- Comment #21 from Willie Walker (reporter, orca developer, points: 20) 2007-11-09 17:44 UTC [reply] Mike believes the fix for this bug will fix a lot of odd interactions he's seeing with tables all over the place, so I'm marking this for 2.21.3 (02-Dec). I think test/keystrokes/gtk-demo/role_tree_table.py will probably also need some new tests added to it: 1) Navigation via Ctrl+{Up,Down} to move the dotted rectangle 2) Shift+{Left,Right} to expand/collapse both unselected and selected nodes 3) Ctrl+Space to select/unselect a node 4) {Up,Down} from an unselected node ==========================
Will sent me some information via email. Adding it to the bug so it doesn't get lost. We need to work out the following: "Here are the discrete things a user can do, and here's the result in the GUI. What should the speech/braille presentation be at each step?" The initial challenge is to come up with the discrete things: * Simple navigation (arrowing, page up/down) that cause both the dotted rectangle (active descendant?) and the selection to change. * Ctrl+navigation to move the dotted rectangle, but not change selection. * Shift+navigation extend/reduce selection either one thing at a time by arrowing or chunks at a time by paging. This also moves the dotted rectangle. * Ctrl+space to change the selection of a single object, but not move the dotted rectangle. * Some other action to change the state of the object where the dotted rectangle is. For example, Shift+{Left,Right} on a tree node to expand/collapse it. Note that I don't think this is going to be an easy problem to solve. When I worked on the original code, I had no idea the multiselect capabilities even existed and conflated the dotted rectangle and selection into one thing. From the code side, I think this equates into treating active descendant and selection changes for "manage descendant" kind of things as the same. So, in default.py, we have onActiveDescendantChanged which I think is what is handling the dotted rectangle. Then, we have onSelectionChanged, which short circuits the selection for things that manage their descendants. The onSelectionChanged is probably the thing that needs work.
Created attachment 99685 [details] Evaluation for two Gtk+ widgets. Here's the current situation w.r.t. the various keystrokes, for two types of Gtk+ widget: 1/ Layered (Icon) Pane. 2/ Tree Table. Note these tests are done from Orca SVN trunk, with the patch (http://bugzilla.gnome.org/attachment.cgi?id=99055) for bug #486908 applied. Mike, for each test case, is the braille and speech output correct? If not, what needs to change? Please be specific: what new information needs to be presented, what exactly should be brailled/spoken and where should that come w.r.t. the other items that are presented. Will, there appears to be a bug (look for "BUG?"). Do you agree? If so, I'll file it against atk/gail. Thanks.
> Will, there appears to be a bug (look for "BUG?"). > Do you agree? If so, I'll file it against atk/gail. I see this in the file: "When Control-Space is pressed, we do not get any events. [BUG?] This would seem like a bug, otherwise we have no way of detecting that the selection state for this icon has changed." This definitely looks like a bug -- I'd expect to see appropriate "object:state-changed:selected" events for things that were selected and unselected as the result of pressing Ctrl+Space. When I used both accerciser and orca to look for these events in the Icon View Basics demo, I saw no "object:state-changed:selected" events when I pressed Ctrl+Space.
Bug #499835 - No "object:state-changed:selected" event when [un]selecting icon in layered pane.", has been opened against atk/gail for that problem.
(In reply to comment #14) > Mike, for each test case, is the braille and speech output correct? > If not, what needs to change? Please be specific: what new information > needs to be presented, what exactly should be brailled/spoken and > where should that come w.r.t. the other items that are presented. Still curious if Mike needs to answer this -- Rich, with the exception of the blocking bug, do you have the information you need? Rich, if you do get the information you need, how much of this bug can you address even if the blocking bug isn't fixed?
> Still curious if Mike needs to answer this -- Rich, with the exception of the > blocking bug, do you have the information you need? No. Mike still needs to respond (see comment #14). > Rich, if you do get the information you need, how much of this bug can > you address even if the blocking bug isn't fixed? Unclear until I know what Mike wants to do.
(In reply to comment #12) > Several comments related to the selection and navigation > problems were added to bug #486972. Copying them here so > that they are all in one place. > > ------------------- > > Comment #15 from Rich Burridge (orca developer, points: 21) > 2007-11-06 21:48 UTC [reply] > > We discussed this bug in the Orca team meeting today. For the > problem of not reporting anything when we initially Down arrow > into the table, we are going to try out the following approach > to see if it will fix the problem. (Note that if you Down arrow > a second time, it will select that first entry.) > > If we are now selected but weren't previously, then we should > speak/braille the expanded/collapsed state. > > Returning bug state to normal while this is investigated. > > ------------------ > > Comment #16 from Willie Walker (reporter, orca developer, points: 20) > 2007-11-07 19:02 UTC [reply] > > Mike - I think we need some design input from you. This may turn into a > separate bug, but the new situation is that the table in question has the > following multiselect behavior as brought to our attention by Joanie: > > * A dotted rectangle around the row lets you know about the current cell of > interest. There's only one dotted rectangle. Moving the dotted rectangle up > and down without changing selection is managed via Ctrl+{Up,Down}. When this happens we should simply speak the item with the rectangle around it. If the item has the rectangle and is also selected we should speak the item followed by selected. > > * The highlighting of a cell indicates if it is selected or not. There can be > multiple discontiguous sets of cells selected. The toggling of selection for > the current cell of interest is managed via Ctrl+Space. Selection is also > implicitly done when just Up or Down are pressed. When just Up or Down are > pressed, all selection is cleared and then the newly landed on thing (i.e., the > thing that gets the dotted rectangle) becomes the sole thing selected. > If CTRL+space is pressed we should simply speak selected or unselected. > We need your user experience design on this. In particular, I think a general > model what should be presented when: > > * Navigating the dotted rectangle to a cell -- I'm assuming the contents should > be spoken following the "read table cell or entire row" setting. But, I'm > assuming the indication of whether the thing is selected or not is also > important and should be presented. What should the presentation be in speech > and braille for this and under what conditions should the presentation be done > (e.g., only when unselected?). > See my above comments on the expected speech output. For braille we should honor the users braille selection settings. ie if the item is selected put dots 7 and 8 under it. > * Expanding/collapsing a cell that's not selected/highlighted. I'm assuming > something should be spoken here, and it should match the output we get when we > expand/collapse the node when it is selected? > If the state is changed simply speak the state change. There is no reason to speak the item again. > * Changing the selection state of the cell. I'm assuming we should also > present something here. What should the presentation be in speech and braille > for this? > See the above for pressing CTRL+space.
Mike, thanks for your comments, but I still need some clarification here. You say: > When this happens we should simply speak the item with the rectangle around it. But what should be spoken? It's not a selection, so what do you want to say and braille to the user to differentiate this from an item that just has focus?
Also Mike, could you please take a look at the various scenerios I outlined in detail in the attachment in comment #14. Here's the link: http://bugzilla.gnome.org/attachment.cgi?id=99685&action=view What would help me tremendously is if you could give Mike, what would help me tremendously is if you could for each test case, tell me if the braille and speech output is correct? If not, what needs to change? Please be specific: what new information needs to be presented, what exactly should be brailled/spoken and where should that come w.r.t. the other items that are presented. Thanks.
> > But what should be spoken? It's not a selection, so what do you want to > say and braille to the user to differentiate this from an item that just has I'm thinking that if the item has focus but is not selected we should speak "not selected at the end of the speech. If the item has focus and is selected we should speak "selected" at the ed of the speech string. I'm thinking of some ways to use our verbosity settings here as well and will write more shortly. > focus? >
Mike, I believe the information needed from you is as follows (from comment #21): > Also Mike, could you please take a look at the various scenerios > I outlined in detail in the attachment in comment #14. Here's the link: > > http://bugzilla.gnome.org/attachment.cgi?id=99685&action=view > > Mike, what would help me tremendously is if you could for each test case, > tell me if the braille and speech output is correct? > > If not, what needs to change? Please be specific: what new information > needs to be presented, what exactly should be brailled/spoken and > where should that come w.r.t. the other items that are presented.
*** Bug 509834 has been marked as a duplicate of this bug. ***
Here's another test case from bug #509834: Steps to reproduce: Prep before running the actual test. 1. Open your home directory in nautilus. Set it to list view. 2. Down arrow to the first list entry in your home directory and press Return. Make sure that that directory is also being shown in list view. 3. Quit that nautilus window. Steps to reproduce: 0. Start Orca. 1. Open your home directory in nautilus in list view. 2. Down arrow to the first list entry and press Return 4. Down arrow to the first list entry. After each of those down arrows, the first list entry is selected but not focused.
Created attachment 103461 [details] Test cases with my comments. This is my comments for the above test cases.
Created attachment 103463 [details] Adjusted version, to easily see where Mike has added comments.
Just spoke with Mike. W.r.t. tree table cases #1 and #2, both should speak: "new years day tree level 2 selected" at the end.
Spoke to Mike again. The adding of "selected" would be in the _getSpeechForTableCell() method in speechgenerator.py. Mike would like "selected" spoken for all table cells (list or tree) if they are selected. It should also be the last thing spoken. Note that this will affect some our regression tests.
Created attachment 103502 [details] [review] Potential fix for tree table cases #1 and #2 - Revision #1. This patch now adds "selected" to the end of the table cell(s) utterances if the table cell that currently has focus is also selected. Note that the "tree level n" utterance (if present) is after this because that is an utterance that is determined in locusOfFocusChanged() in default.py, and occurs after the speech utterances for the table cell(s) has been collected. If this *really* does have to go at the end of the utterances, then I'll need to move the code out of _getSpeechForTableCellRow() and special case it in locusOfFocusChanged(). Mike, please try what I've currently got and let me if you *really* want this at the very end. Note that you only get a "tree level n" when it changes tree levels, so (for me) it's more memorable hearing it at the end. By my understanding of what Mike wants (modulo the difference stated above), this I believe fixes the tree table cases #1 and #2 in the sel_eval.txt document. As has been mentioned before, it also affects what is spoken whenever there is a table cell with focus. If this is want we want, then certain regression tests will need to be changed again. I believe the other sel_eval.txt problems (what we should say when the user does Control-space in a layered icon pane) will require bug #499835 to be fixed. Note also that adding a fix for uttering "selected" for table cells in StarOffice.py and Evolution.py hasn't been attempted yet. I'd like to get consensus for the various gtk-demo table cell demos before I work on that. Please test.
While I'm waiting for some regression tests to run on my development box, I figured I'd give this a spin. I'm afraid this feels a bit verbose to me. Without reading back through every last comment on this and the related bugs, but going strictly by my admittedly faulty memory, I thought the problem we were trying to address is how best to present the various oddities that occur in multi-select environments -- oddities that one can immediately see visually, such as the fact that a given item which was arrowed to is not selected/highlighted like one would expect. If not, please ignore the following. :-) Given this assumption, I wonder if the approach we might want to take instead is one similar to what we do (recently revised/refined) in Evolution's message list, namely present the user with the info that differs from the norm -- and do so with as little verbiage as possible. When navigating up and down in a tree or tree table, the norm is that the item is selected. The "non norm" is that the item is unselected. I think we're already doing the "norm" bit (aren't we?). The trick is how to present the non-norm. I would therefore suggest the following: 1. Don't say "selected" all the time as the patch pretty much does. It's not relevant when moving by column, and 9 times out of 10, it's not relevant when moving by row. Selected is the norm. 2. Consider saying "not selected" when focus is initially given to a tree/tree table and the item with focus is not selected. That's the non-norm. By saying "not selected" after the item's name ("January not selected") we're conveying the oddball state -- and also preparing the user for what is to follow should Down Arrow be pressed and we're still in January, which brings me to: 3. DO say "selected" (or "unselected") if the current locusOfFocus is the same, but the state has changed. i.e. just like we do with checkboxes whose state gets toggled. Thus the first Down Arrow would result in "January not selected". The second Down Arrow would keep us in January (how odd!), thus "selected." This would also address the case where the user toggles the selected state of the current item by pressing Control+Space. Again, like checkboxes. If the above is deemed worthy of implementation: 4. Consider providing this information earlier in the process than at the very end. If I Down Arrow into January and Orca tells me I'm in January and I have no interest whatsoever in January, I'm not going to wait until Orca finishes its explanation of where January happens to be placed w.r.t. its parent or what tree level it happens to be at. Instead, I'm going to Down Arrow again because I'm in a tree and that's what one does in a tree to locate the next item. As a result, I'm going to miss out on the "not selected" at the end. Hearing: "January not selected" (Down Arrow) "Selected" (Down Arrow) "New Years Day" means I get the information/explanation I need immediately and succinctly and can move on that much faster. For whatever it's worth.... Back to those regression tests. <smile>
Thanks Joanie. Okay, so this is different than what Mike commented on in the sel_eval.txt document and what I understood him to say in subsequent phone conversations, so I think we need our UI guy to tell us what he really wants. Mike?
It was actually Will that felt "selection" should always be spoken when navigating up and down in a tree table. I'm going to the users list with this one.
> > 1. Don't say "selected" all the time as the patch pretty much does. > > It's not relevant when moving by column, and 9 times out of 10, > it's not relevant when moving by row. Selected is the norm. this is fine with me.> > 2. Consider saying "not selected" when focus is initially given to > a tree/tree table and the item with focus is not selected. > This is fine as well. > 3. DO say "selected" (or "unselected") if the current locusOfFocus > is the same, but the state has changed. > This is already planned. It is waiting on a bug that Li is working on. > 4. Consider providing this information earlier in the process than > at the very end. > > If I Down Arrow into January and Orca tells me I'm in January and > I have no interest whatsoever in January, I'm not going to wait > until Orca finishes its explanation of where January happens to be > placed w.r.t. its parent or what tree level it happens to be at. > Instead, I'm going to Down Arrow again because I'm in a tree and > that's what one does in a tree to locate the next item. As a > result, I'm going to miss out on the "not selected" at the end. > Hearing: > > "January not selected" > (Down Arrow) > "Selected" > (Down Arrow) > "New Years Day" > > means I get the information/explanation I need immediately and > succinctly and can move on that much faster. > I'm not personally in favor of moving the state to the beginning but I can go along if everyone else disagrees.
I defer to our UI leads decisions. :-)
Created attachment 103649 [details] [review] Hopefully a fix for tree tables based on comments #31 and #34 Note that this won't fix the gtk-demo icon pane problems. That's going to need a fix to bug #499835. This should produce the desired speech output for gtk-demo Tree View demo's: Tree Store and List Store. Please test.
Created attachment 103735 [details] [review] Fix for Control-Space [un]selection in icon layered panes as well. This version of the patch (hopefully) also correctly handles icon selection and unselection via Control-Space as well. You will need the following patch from bug #499835: http://bugzilla.gnome.org/attachment.cgi?id=103721
Mike, I think it's time for you to start testing this patch: http://bugzilla.gnome.org/attachment.cgi?id=103735 to make sure it meets all your requirements for selection in icon layered panes and tree tables. You will also need to apply the latest patch from bug #499835: http://bugzilla.gnome.org/attachment.cgi?id=103721 to the gtk+ module. As I tested this on Ubuntu Gutsy, I just downloaded the tarball that is the equivalent to what's installed on my system (minus the Ubuntu specific changes): http://ftp.gnome.org/pub/GNOME/sources/gtk+/2.12/gtk+-2.12.0.tar.gz and applied the patch to that. You will probably also need to do a: % sudo apt-get build-dep libgtk2.0-0 as well.
For some reason with this patch applied I always hear "unselected" before each item when I arrow between buddies in pidgin.
Noted. I'll look at that on Monday. Mike, does it do what you want for the gtk-demo tree and list store demos? How about the gtk-demo Icon Pane basics demo? Thanks.
Rich, For what it is worth, i am a blind user and totally agree with comment 31. in terms of selectedness: I feel the order should be <objecttext> <selected> <hierarchy> and when several objects are being added in one go to the selection then lett us know which once are being added. We obviously dont need to hear which once has already been selected, just the most recent additions. example (angle brackets = selected, round brackets = selection box): <a> [b] [c] (usual arrow key selected a) [d] [e] [f] we move across to b (ctrl arrow) (<a> [b]) [c] (we output b not selected.) [d] [e] [f] if ctrl+space, we say "selected", we already know its b, so dont need to respeak it) (<a> <b>) if we now shift arrow down (<a> <b>) [c] (<d> <e>) [f] (we say d selected, e selected). if we use control and arrow keys to move around, selected/not selected should be announced for each item, the whole point of navigation in this mode is to 'look' at the current status, so selected/not selected should always be spoken. on the other hand if we simply use the arrow key to move to f then we announce "f" without any selection status. There are tree structures with selection issues? sorry i havent tried the patches so cant comment there.
Hi Jon. Thanks for your feedback. Hopefully, Mike, our UI guy, will take this into consideration and decide whether anything should be changed. > There are tree structures with selection issues? See the first couple examples in the attachment: http://bugzilla.gnome.org/attachment.cgi?id=99685
There's a correction on the pre-requirements for testing this patch. You need both the first and last patch to bug #499835, plus the latest patch to this bug, in order for this to work. Thanks to Joanie for confirming this. So here it is in more detail: You will need the latest patch from this bug: http://bugzilla.gnome.org/attachment.cgi?id=103735 to make sure it meets all your requirements for selection in icon layered panes and tree tables. You will also need to apply two patches from bug #499835: http://bugzilla.gnome.org/attachment.cgi?id=103319 http://bugzilla.gnome.org/attachment.cgi?id=103721 to the gtk+ module. As I tested this on Ubuntu Gutsy, I just downloaded the tarball that is the equivalent to what's installed on my system (minus the Ubuntu specific changes): http://ftp.gnome.org/pub/GNOME/sources/gtk+/2.12/gtk+-2.12.0.tar.gz and applied the patch to that. You will probably also need to do a: % sudo apt-get build-dep libgtk2.0-0 as well. Thanks.
I applied the relevent patches on Saturday. So far this seems to be working much better. The pidgin problem from my previous comment does exist both with and without the gtk patches though.
Created attachment 103897 [details] Orca debug output from testing pidgin with this patch. Joanie noticed a similar problem with Evolution on Friday. Here's what's happening. I can see the same in the attached debug.out (about line 326), but I'm not hearing it. In other words, some more speech (the utterance at line 680) is coming along and stopping the previously queued utterance of "unselected". In pidgin, in the Buddy window, when I press Down (line 296), we get an "object:selection-changed" event (line 311): vvvvv PROCESS OBJECT EVENT object:selection-changed vvvvv OBJECT EVENT: object:selection-changed detail=(0,0) app.name='pidgin' name='None' role='tree table' state='enabled focusable focused sensitive showing visible manages descendants' relations='' At that point in time, the object with focus has changed selection state so the code is causing "unselected" to be spoken. ... Very soon after that, we get an "object:active-descendant-changed" for the selection (line 605): vvvvv PROCESS OBJECT EVENT object:active-descendant-changed vvvvv OBJECT EVENT: object:active-descendant-changed detail=(9,0) app.name='pidgin' name='None' role='tree table' state='enabled focusable focused sensitive showing visible manages descendants' relations='' and that causes the following utterances (line 680): SPEECH OUTPUT: '' SPEECH OUTPUT: 'mmpedersen1000' SPEECH OUTPUT: 'tree level 2' So, and Joanie suggests, we need to refine the patch to the onSelectionChanged() method in default.py, so that it doesn't speak these "bogus" "unselected" utterances. Her suggestion was perhaps we only care about these "unselected" utterances, if the number of selected items is not 1. My concern here is how do you know whether this "object:selection-changed" was a selection or an unselection? In other words, you could test a tree table and find it's now got one selection, but is that because: a) we have a selection event (and previously there was nothing selected). b) we have an unselection event (and previously there were two items selected). Any and all suggestion welcome on how to refine this.
I took a look at this over lunch. Seems that in tree tables (at least in the case of Pidgin and Evo) if the active descendant changes, we get an object:selection-changed event for the unselection of the descendant that's giving up focus. I'm not seeing such an event for the selection of the newly activated descendant. I think this is inconsistent. What's the definition of a selection-changed event? Either we should get events for the item that's giving up focus AND the item that's gaining it (not my preference), or we should not get events when the cause of the selection changing is the active descendant changing (my preference as this selection-changed event in this case is redundant). If we only got selection changed events when the focus/active descendant were not changing, I think we'd be doing the right thing with your existing patch. Perhaps Li could cause this to be fixed on the Gtk side of things?? With your patch in nautilus, we say "selected" every time as we move from icon to icon. We didn't used to do that. I personally don't think we should be doing it now. In both the icon panel demo and in Nautilus, when focus is moved to an item with Control+Arrowing, the newly focused item is not selected. With trees and tree tables, your patch causes us to indicate the fact that things are not selected. Should we be doing the same thing with the icon panels? (Note that we don't say it's selected in this case, so one *can* distinguish the conditions. But we're presenting the norm rather than the "non-norm.")
(In reply to comment #46) > I took a look at this over lunch. Seems that in tree tables (at least in the > case of Pidgin and Evo) if the active descendant changes, we get an > object:selection-changed event for the unselection of the descendant that's > giving up focus. I'm not seeing such an event for the selection of the newly > activated descendant. There are two sources of events to think about here: 1) the parent that manages the selection, and 2) the children that are selectable. It seems we should get object:selection-changed events for parents that allow children to be selected (i.e., they support the Accessibility::Selection interface), and they should be emitted whenever the selection changes. When an object itself becomes selected or unselected, it should emit an object:state-changed:selected event with a detail1 that is True or False, indicating if it is selected or not. In addition, the active descendant may very well be the selected descendant, but I think the two can become decoupled when doing things such as Ctrl+{Up,Down} to move the dotted rectangle to a new active descendant, but keeping the selection the same.
I'm not sure I follow you. If the selection changes because the focus has changed, you are suggesting that we are missing the object:select-changed event for the newly selected item? > When an > object itself becomes selected or unselected, it should emit an > object:state-changed:selected event with a detail1 that is True or False, > indicating if it is selected or not. Yup, we get plenty of those. :-) Perhaps that is what we should (also) be attending to? > In addition, the active descendant may very well be the selected descendant, > but I think the two can become decoupled when doing things such as > Ctrl+{Up,Down} to move the dotted rectangle to a new active descendant, but > keeping the selection the same. Not sure, but I believe Rich is handling that in locusOfFocusChanged(). The problem is the extraneous "unselected"s that are spoken as a result of getting an object:state-changed:selected event when an object is giving up focus -- which occurs prior to our being notified that the active descendant has changed. (i.e. we think we're still focused). Definitely an argument in favor for the grouping/clustering of events. :-)
(In reply to comment #48) > I'm not sure I follow you. If the selection changes because the focus has > changed, you are suggesting that we are missing the object:select-changed event > for the newly selected item? The following is talking in the "general" sense... Selection and focus are two separate things. When selection changes, we should get object:selection-changed for the parent and object:state-changed:selected for the children. If we don't get events for the parent and for the children whose selection state changed (i.e., they were selected or unselected), then this seems like a bug. When focus changes, we should get focus: and object:state-changed:focused events. If we don't get events for these, then it might be a bug. The reason I say *might* is that we could be dealing with a parent who is managing descendants. In those cases, we won't get focus events, but instead should get active-descendant-changed events for the parent. When one arrows up and down in these beasts we're talking about, *both* the selection *and* one of {focus,active descendant} are changing, so we should be getting events for both.
Just to give an example of things that we probably should expect to see in these beasts. For this example, just assume the thing we're dealing with is a beast that manages descendants. 1) User arrows up/down: we should both selection and active descendant events, the selection should ultimately end up at 1. 2) User shift+arrows up/down: we should both selection and active descendant events. The selection count increments/decrements depending if the selection was extended or reduced. 3) User presses Ctrl+space: only the selection changes 4) User ctrl+arrows up/down: we should only get active-descendant events. The selection stays the same, but the active descendant changes. If we're dealing with a parent that does not manage its descendants, we should get focus events instead.
(In reply to comment #50) s/we should both/we should *get* both/ Sorry...
Thanks for the comments Will. I'm going to try adding in a new listener for "object:state-changed:selected", to call onStateChanged(), and move the new chunk of code from onSelectionChanged() to onStateChanged() and adjust appropriately. Hopefully I'll have a new patch to try in a little while.
Created attachment 103914 [details] [review] Patch to hopefully improve things. This patch adds a listener in default.py for "object:state-changed:selected" events, and calls onStateChanged() when one is found. The logic for uttering "unselected" or "selected" that was in onSelectionChanged(), has now been moved to onStateChanged() and adjusted appropriately. Note that you really only need the last attachment to bug #499835: http://bugzilla.gnome.org/attachment.cgi?id=103721 as well as this patch, to get this working. Evolution and pidgin behavior are hopefully back to their old selves. The gtk-demo icon pane, tree store and list store demos seem to be working correctly. I'm sure there is still some more tweaking to do, but this seems to be much better. Please test.
I still get the "unselected"s in Pidgin and Evo. In Pidgin, they are more quickly interrupted it seems, but still present. In Evo I hear them more frequently than before. I also hear unselected when I press Return on any of the items in the the gtk-demo list of demos. We're no longer saying "selected" for each item in Nautilus (yea!), but we've started saying it for each item in the gtk demo icon panel. (And, yes, I removed the extra patch I thought had been necessary :-) )
I'm seeing the same results as Joanie with the latest patch.
Okay, thanks for testing. I think it's time for Will to advice on the best way of handling/fixing this.
Created attachment 103967 [details] [review] Alternative version of the patch. Here's a new version to try. If we get an "object:state-changed:selected" event, it now checks to see if the last user input was a Control-Space keyboard event. If so it'll say "selected" or "unselected" as appropriate. I believe this now gets rid of the bogus "unselected" utterances in Evolution and pidgin. Note that Control-Space in pidgin does cause "object:state-changed:selected" events to be generated, but they look like this: vvvvv PROCESS OBJECT EVENT object:state-changed:selected vvvvv OBJECT EVENT: object:state-changed:selected detail=(1,0) app.name='pidgin' name='None' role='table cell' state='enabled focusable selectable selected sensitive showing transient visible' relations='' ^^^^^ PROCESS OBJECT EVENT object:state-changed:selected ^^^^^ In other words, there is no "focused" state. Seeing as I'm using that to trigger the speaking of "selected" or "unselected", we are not hearing that for pidgin. Does this get us closer to what we want?
I'm still testing this but so far I'm no longer getting the unwanted unselected messages in pidgin and the application launch menu. Much better so far
> in terms of selectedness: > I feel the order should be > <objecttext> <selected> <hierarchy> > I personally prefered to have the output the way it is implemented in the latest patch with the state spoken last. That being said if the community feels strongly that the information should be reversed we can always change it. Lets see what the feeling is after the coommunity at large starts testing this functionality and giving us feedback. > and when several objects are being added in one go to the selection then lett > us know which once are being added. > We obviously dont need to hear which once has already been selected, just the > most recent additions. Just to be sure I'm understanding your comment here: Are you refering to selection in icon views? thanks much for the comments.
Created attachment 103983 [details] [review] Hopefully the final revision. Mike found a case I wasn't handling with Control-[Up,Down,Left,Right] to another icon in the gtk-demo Icon View Basic demo. Hopefully this patch now correctly handles that and hasn't broken anything else.
I think this patch is looking good. Perhaps it should now get checked in for use by the wider user community. If we decide based on user feedback that tweaks still need to be made perhaps those can be separate bugs. Thanks a lot Rich, this has been a bit of a pain but I think we've come up with something really useful.
From a code review standpoint, this looks good to me as well. If it pylints and regression tests well, I say commit and be done with this beast. :-)
Patch committed. Moving to "[pending]" state. Thanks. It's pylinted. I'll run the regression tests now.
Created attachment 103996 [details] [review] Changes to the regression tests. * test/keystrokes/gtk-demo/role_icon.py: test/keystrokes/gtk-demo/role_tree_table.py: Adjusted the regression tests for the gtk-demo icon and tree tables, for the tests that the changes for bug #486908 have fixed.
Sorry, i dont know what i seem to be doing wrong, this is what im experiencing: running vanilla rev 3527. in Nautilus, shift+arrow does not announce items being selected. Same goes for items on the desktop. in gtk-demo icon view -> basic: everything says unselected, even if im just arrowing around. Shift+arrow still says unselected for every item. if i understood your last comment regarding gtk-demo, only patch 103721 is required. Thank you
In testing this patch (and the Gtk+ one) yesterday, Mike and I noted that [un]/selecting the Nautilus desktop icons with Control-Space did not generate any "object:state-changed:selected" events, and therefore Orca couldn't do the right thing. I've filed bug #513213 against Nautilus on this problem.
(In reply to comment #65) > Sorry, i dont know what i seem to be doing wrong, this is what im experiencing: > running vanilla rev 3527. > in Nautilus, shift+arrow does not announce items being selected. Same goes for > items on the desktop. Because the behavior in gnome for holding down shift with any up and down arrow movement always seems to select the current design is not to speak selected as unless we speak unselected or not selected selection should be assumed. As Rich commented we did discover the desktop problem yesterday and hopefully that problem can be fixed for the next gnome release. > in gtk-demo icon view -> basic: > everything says unselected, even if im just arrowing around. > Shift+arrow still says unselected for every item. HMMMM Rich and I talked earlier and neither of us can reproduce this problem. > if i understood your last comment regarding gtk-demo, only patch 103721 [edit] is > required. > You do need the last patch to gtk+. I don't know if it makes a difference but I am also running the latest trunk versions of at-spi, atk, and gail. What distro are you running? > Thank you >
Adding in the gtk+ bug that we depend upon before this gets really fixed.
The latest patch to the underlying bug has just been committed. Finally closing as fixed. If we find more selection problems, then just open new bugs against each one. Thanks.