GNOME Bugzilla – Bug 466251
Support ARIA live regions in Firefox/Gecko
Last modified: 2007-12-11 17:05:50 UTC
The Mozilla developers recently released a substantial change/upgrade to their live region interface https://bugzilla.mozilla.org/show_bug.cgi?id=390692 . This will be viewed as a keystone event and be the starting point to build live region support in Orca. Resources: Mozilla Development Center on live regions: http://developer.mozilla.org/en/docs/AJAX:WAI_ARIA_Live_Regions Mozilla Development Center live region API support: http://developer.mozilla.org/en/docs/AJAX:WAI_ARIA_Live_Regions/API_Support Mozilla Development Center ARIA live region section: http://developer.mozilla.org/en/docs/Accessible_DHTML#Live_regions Test cases: http://accessibleajax.clcworld.net/
Firefox live region attributes are not being persisted. Here is their bug https://bugzilla.mozilla.org/show_bug.cgi?id=392061
Created attachment 93730 [details] [review] first version of Support ARIA live regions in Firefox Very rudimentary first version, mainly placed here for safe keeping. However, the patch does work on the 'live' test cases here http://accessibleajax.clcworld.net/ . Notes: - Utilizes the gobject event loop to 'pump' out queued live region related messages. - Uses priority queue to handle 'live' priority queuing and to maintain chronological order. - MSG_KEEPALIVE_TIME is used to purge queue of old messages. Currently set to 5 seconds, but should be a user setting. - Only outputting text that is held by the text-changed event.source. Will need to add labeling and description info. May need to recurse the tree. - On children changed live region events are not currently handled.
Created attachment 94129 [details] [review] second version of Support ARIA live regions in Firefox/Gecko This version is still very immature and represents only a base to work from. Changes from the last version include the following: 1) added keybinding 'y' to change live region channel. Currently supporting an 'off' and 'default' channel. Live regions will probably add a channel property in the future. This is in anticipation of that. If channels are not added to the live region spec, this can easily be changed to off/on feature. 2) changed gobject.idle_timeout() to gobject.idle_add(). Live region performance improved and the overall performance of Orca did not change very much.
> This version is still very immature and represents only a base to work from. It's definitely looking interesting. > 1) added keybinding 'y' to change live region channel. Currently supporting an > 'off' and 'default' channel. Live regions will probably add a channel property > in the future. This is in anticipation of that. If channels are not added to > the live region spec, this can easily be changed to off/on feature. I would opt for not adding this binding until channel properties are actually added. > 2) changed gobject.idle_timeout() to gobject.idle_add(). Live region > performance improved and the overall performance of Orca did not change very > much. Cool. At some point, I think some more intelligence should be added to the code to only have the idle handler in effect when there is something on the queue. The reason for this is that I believe I've that idle handlers can cause some CPU usage increase. In comparing the code to the docs at http://developer.mozilla.org/en/docs/AJAX:WAI_ARIA_Live_Regions, I have a couple comments regarding interpreting the docs. The spec was a little ambiguous to me. My interpretation was: live="polite": speak only if speech is not busy. If speech is busy, toss live="polite" stuff on the floor. live="assertive": queue up assertive messages. Play the queue when speech is available. live="rude": flush all queues, kill all speech, and speak the rude message right away. I'm guessing we're viewing the FireVox implementation, however, as the interpretation of the spec: live="polite": if speech is busy, queue up polite messages. Play the queue when speech is available. live="assertive": flush any polite messages from the queue and queue up assertive messages. Play the queue when speech is available. live="rude": flush the queue and add the rude message. Play the queue when speech is available. live="rude" seems like it should not only flush the queue, but should also stop all other speech in progress (e.g., speech.stop()). What is the right interpretation of this?
> > 1) added keybinding 'y' to change live region channel. Currently supporting an > > 'off' and 'default' channel. Live regions will probably add a channel property > > in the future. This is in anticipation of that. If channels are not added to > > the live region spec, this can easily be changed to off/on feature. > > I would opt for not adding this binding until channel properties are actually > added. In the degenerate case the key binding serves as an on/off switch. I think this switch is needed in some form or another. I think a user preference, either on the GUI or in user_settings, would not provide the usability needed. > > > 2) changed gobject.idle_timeout() to gobject.idle_add(). Live region > > performance improved and the overall performance of Orca did not change very > > much. > > Cool. At some point, I think some more intelligence should be added to the > code to only have the idle handler in effect when there is something on the > queue. The reason for this is that I believe I've that idle handlers can cause > some CPU usage increase. Cool. I think you are correct. I'll experiment. > > > In comparing the code to the docs at > http://developer.mozilla.org/en/docs/AJAX:WAI_ARIA_Live_Regions, I have a > couple comments regarding interpreting the docs. The spec was a little > ambiguous to me. My interpretation was: > > live="polite": speak only if speech is not busy. If speech is busy, toss > live="polite" stuff on the floor. > > live="assertive": queue up assertive messages. Play the queue when speech is > available. > > live="rude": flush all queues, kill all speech, and speak the rude message > right away. > > I'm guessing we're viewing the FireVox implementation, however, as the > interpretation of the spec: > > live="polite": if speech is busy, queue up polite messages. Play the queue > when speech is available. > > live="assertive": flush any polite messages from the queue and queue up > assertive messages. Play the queue when speech is available. > > live="rude": flush the queue and add the rude message. Play the queue when > speech is available. > > live="rude" seems like it should not only flush the queue, but should also stop > all other speech in progress (e.g., speech.stop()). What is the right > interpretation of this? > I agree with your final assessment of rude messages. I will get clarification from Charles and Aaron on this matter. IMHO, rude messages should flush the queue, stop speech and output the rude message if live regions is not off. This is not the way I have it implemented at the moment. My next patch will address this.
(In reply to comment #5) > > > 1) added keybinding 'y' to change live region channel. Currently supporting an > > > 'off' and 'default' channel. Live regions will probably add a channel property > > > in the future. This is in anticipation of that. If channels are not added to > > > the live region spec, this can easily be changed to off/on feature. > > > > I would opt for not adding this binding until channel properties are actually > > added. > > In the degenerate case the key binding serves as an on/off switch. I think > this switch is needed in some form or another. I think a user preference, > either on the GUI or in user_settings, would not provide the usability needed. OK - so, we'll need to get a call from Mike on this. Mike, what should the desktop/laptop bindings be for enabling/disabling the presentation of live regions? In addition, I'm also curious about what we should do for braille. Mike, do you have suggestions for that?
> In addition, I'm also curious about what we should do for braille. Mike, do > you have suggestions for that? > This leads me to think about verbosity settings. Settings could affect labels, descriptions, live region IDs and others information perhaps. Braille would follow speech, possibly including a live region id.
Created attachment 94284 [details] [review] third version of Support ARIA live regions This version is looking even better. It follows the discussion Will and I had earlier about rude live regions. Here is an interesting test case. http://accessibleajax.clcworld.net/advanced/live_multiple-regions.htm
Created attachment 95476 [details] [review] fourth version of Support ARIA live regions in Firefox/Gecko This version adds the following: 1) Supports new AT-SPI event name format. This will have to be looked at when http://bugzilla.gnome.org/show_bug.cgi?id=475177 is committed. 2) Adds support for object:children-changed:add events. 3) Adds support for atomic property. 4) Adds support for labelledby. Future work: 1) support 'relevant' property. 2) support 'controls' property. We may get this one for free. 3) support 'describedby' property. We may need verbosity settings. 4) Filter out repeated announcements, possibly by looking for an embed character in a text event. In this case, an object:children-changed event will follow. The following are the the event scenerios. child removed + text remove and child added + text added text remove + child added + text added text remove + text added child remove + text remove and text added The events depend on whether the old text had its own object and whether the new text gets its own object. 5) Support remove events.
Created attachment 97015 [details] [review] fifth version of Support ARIA live regions in Firefox/Gecko This is the first version after our discussion in Boston. Some of the changes include the following: 1) Removal of all channel logic. 2) Addition of liveRegionsOn constant in Gecko.py to turn live region support on/off. 3) Addition of user commands 'shift-r' and 'r' to move to prev/next live region. 4) Addition of user command to advance the live region politeness level. The order is the following off->polite->assertive->rude->off .... The command is currently bound to 'y' until a more suitable key command is determined. Future work will include hooking the politeness level into the bookmark feature.
Mike, Give this new version a try with http://accessibleajax.clcworld.net/advanced/live_multiple-regions.htm . There are four live regions marked off, polite, assertive and rude. Try the user settings I described and let me know if I am on the right track. You may wish to switch the update time to a higher value by using the combobox at the top of the page. Would a 'turn all live regions off' command be useful?
Created attachment 97144 [details] [review] fifth version of Support ARIA live regions in Firefox/Gecko (corrected) Corrected version after I forgot to do svn add liveregions.py after I rebuilt my system.
I've just found a pretty nasty problem with ths patch. After applying the patch do the following: 1. open this bug in bugzilla. 2. navigate to the new commenn text entry field. 3. attempt to type an "r" You will find that the "r" is not entered into the text field. Instead orca starts searching for live regions. The intended functionality here seems to be working well on the test site. It would really help if you could find and include some more test cases though.
(In reply to comment #13) > I've just found a pretty nasty problem with ths patch. After applying the > patch do the following: > 1. open this bug in bugzilla. > 2. navigate to the new commenn text entry field. > 3. attempt to type an "r" > You will find that the "r" is not entered into the text field. Instead orca > starts searching for live regions. Oops. I forgot to add the methods to the structural navigation array. This has been fixed. > The intended functionality here seems to be working well on the test site. It > would really help if you could find and include some more test cases though. > Yes, I agree. I may have to write my own :(
Created attachment 98185 [details] [review] sixth version of live regions, first version post migration This is the first pyatspi migration version. What is interesting is that the children-changed problem, namely pointing to the wrong child, is not an issue in this version. Does this indicate a caching problem in the old version of Orca? One problem that needed to be overcome was multiple method calls for certain event types. The was accomplished by ensuring that all registrations of a given event type were the same and removing the trailing ':' from the event type. With the object:children-changed event fixed, the chat example seen here http://accessibleajax.clcworld.net/ajax_chat/chat.htm is now supported. Try navigating to it and changing the politeness using 'y'. Mike, have you thought of a better key binding yet?
Lets set this to "\" this will leave all letters available for navigation.
Created attachment 98781 [details] [review] seventh version of live regions This patch extends the bookmark feature by providing for observer callbacks on save and load events. This allows the live region manager to persist politeness level overrides to disk by using the infrastructure within the bookmarks object.
One other issue about the the seventh patch. I had problems binding "\" so the politeness advance is still bound to "y". I'm not quite sure what the problem is. "\\" syntax did not help.
Hey Scott, have you come across any more sites that take advantage of this functionality? This new feature would be really nice to test on more than just a couple samble applications.
Mike, Unfortunately I have not found any new sites. I will work on that and post test cases or links to other validated examples in the near future.
Here is a good test site that contains two live regions. Each live region has a set of controls to change politeness, channel (unsupported at the moment), atomic and relevant (unsupported at the moment) live properties. There is also a setting that controls update times. I find it to be a big help. Also, be aware that there is a Firefox bug that causes events to be fired from other pages. So it could quite verbose if you have multiple pages with live regions on each open.
I think you forgot to paste the link for your previus comment.
I'm such an airhead at times. Sorry Mike. Here's the link http://test.cita.uiuc.edu/aria/live/html.php?title=Live%20Regions%20Example%201&ginc=includes/live1.inc&gcss=css/live1.css&gjs=../js/globals.js,../js/enable.js,js/live1.js
Created attachment 98843 [details] [review] eighth version of live regions Adds some live 'atomic' support. The following URL points to an excellent test site. However, I have noticed times when the setting did not take effect. I am printing out the object attributes so this can be cross referenced. http://test.cita.uiuc.edu/aria/live/html.php?title=Live%20Regions%20Example%201&ginc=includes/live1.inc&gcss=css/live1.css&gjs=../js/globals.js,../js/enable.js,js/live1.js
Created attachment 98982 [details] live region test page This is a live region test case. It is identical to the UIUC test case that was mentioned in previous comments except that an initialization problem has been fixed. The tgz file contains the following files: livetest.html livetestfiles/enable.js livetestfiles/live1.js livetestfiles/valid-html401.png livetestfiles/globals.js livetestfiles/live1.css
An issue has been seen that breaks the user controls for the newly attached test case. I am currently working on the differences between this test case and other test cases that were used during this development. For user control testing try using http://accessibleajax.clcworld.net/advanced/live_multiple-regions.htm as an alternative for now.
Created attachment 98997 [details] [review] ninth version of live regions Notable changes: * Reorganized _getUtterances() to give better support for 'atomic' attribute. * * Fixed user controls for live region with obj attribute 'xml-roles' set. * Partially fixed live region navigation with xml-roles set. Still having problems with lists.
Scott and I chatted about this on IRC. It seems OK, but I think Mike needs to play with this and provide some user interaction comments. In particular: * Should rude events always interrupt speech immediately or should an already speaking rude event be allowed to complete? The problem with immediate interruption, which is what is done now, is that you may never hear a complete message. I'd suspect the desired experience might be similar to what we do with progress bars. * Right now, you need to go to a live region to set its politeness level. With lots of regions on a page, this might be a pain. To help with the user experience, it might be nice to have a couple things the user can do: 1) 'take me to the live region that just caused the last chunk of output' and 2) 'don't let any live region below this politeness level to speak'. Scott, I'm also assuming you will remove the debugging 'print' statements and will also work on the docs for translators before checking things in?
Created attachment 99043 [details] scoreboard test case This is an adaptation of Charles's scoreboard example that works around a Firefox labeling issue. This test is mainly for usability studies and is not intended to be a unit or regression test case.
Created attachment 99052 [details] [review] tenth version of live regions Gives support for the following: 1) liveregions.POLITENESS_LEVEL allows the user to ignore messages that have lower politeness levels. 2) Rude messages do not clip other rude messages. However, rude messages do still clip 'lesser' messages. > * Right now, you need to go to a live region to set its politeness level. With > lots of regions on a page, this might be a pain. To help with the user > experience, it might be nice to have a couple things the user can do: 1) 'take > me to the live region that just caused the last chunk of output' I am concerned about timing being an issue. The 'last' live region may change quickly on a rapidly changing page causing the user to move to the wrong live region. This might be a real pain to implement too.
Created attachment 99093 [details] [review] eleventh version of live regions Notable changes: 1) liveregions.POLITENESS_LEVEL (as explained in previous comment) has been removed. 2) advance live politeness level key binding has been changed to '\' 3) monitor live regions user command key binding has been added to 'shift+\' This function works on a toggle. The first press sets all live regions on the page to 'off'. The subsequent press restores all the live region politeness levels either to their bookmarked value or to the HTML markup value if it has never been bookmarked. 4) Politeness bookmarks are not automatically saved to disk when the user advances the politeness. They must explicitly save the bookmarks now.
Scott, to me this looks like the user interface we discussed. Still testing though as these sites present quite a bit of behavior that I haven't really worked with before so I want to give it a lot of testing.
My conversation with Mike yielded three good ideas that I will now try to implement. 1) A user command to hear one of up to 6 previous messages from a given live region. 2) Implement Will's idea of moving to the live region which was just announced. 3) Change output so that rude messages don't interrupt any other utterances.
(In reply to comment #33) > My conversation with Mike yielded three good ideas that I will now try to > implement. > > 1) A user command to hear one of up to 6 previous messages from a given live > region. I view this functionality in one of two ways: 1) up to 6 live messages will be cached. The user will hit one of six key bindings to announce the saved message. Key binding 1 (whatever that may be) announces the previous message, key binding 2 announces the next previous message and so forth. 2) up to 6 live messages will be cached per live region id. The user will first navigate to the live region then hit one of six key bindings to announce the saved message. Key binding 1 (whatever that may be) announces the previous message for that live region, key binding 2 announces the next previous message for that live region and so forth. I opted to partially implement case 1. The key binding is hooked to 'shift+y' and announces the third previous message. I know this is a bit of a hack, but I am running out of key bindings as I'm sure you are aware. Please provide me the key bindings and I will implement either of the two cases you choose. > > 2) Implement Will's idea of moving to the live region which was just announced. > Done. Currently bound to 'y'. > 3) Change output so that rude messages don't interrupt any other utterances. > Changed. Here are the rules. 1) Priority queue based on live politeness setting and in chronological order. 2) Messages older than 5 seconds are dumped. 3) polite and assertive messages are queued. 4) rude messages clear the priority queue of all non-rude messages and are then queued. If they are first in line they end up being spoken immediately as long as no other output is occurring. 5) No message clipping whatsoever. Patch to follow
Created attachment 99215 [details] [review] twelfth version of live regions This patch matches the work described in the previous comment.
> I opted to partially implement case 1. The key binding is hooked to 'shift+y' > and announces the third previous message. I know this is a bit of a hack, but > I am running out of key bindings as I'm sure you are aware. Please provide me > the key bindings and I will implement either of the two cases you choose. I think we need Mike's input on this. It seems as though these messages may be a bit like instant messages, so maybe we should consider providing a user experience similar to what we get with gaim/pidgin (Orca+#). But, that seems to conflict with bookmark stuff. So, I have no good ideas right now. > Changed. Here are the rules. > 1) Priority queue based on live politeness setting and in chronological order. > 2) Messages older than 5 seconds are dumped. > 3) polite and assertive messages are queued. > 4) rude messages clear the priority queue of all non-rude messages and are then > queued. If they are first in line they end up being spoken immediately as long > as no other output is occurring. > 5) No message clipping whatsoever. As long as this works for Mike, it's good by me. I think we might end up needing more user experience with real world pages to see what the user experience is really like. Our current experience led to the above, though, so we're probably very close if not already on the money. :-) For comments on the patch, it looks like we have the following new keybindings: {Shift}+r = navigate to previous/next active region on page y = go to the last/current active region that spoke Shift+y = review the 3rd live region \ = advance politeness level Shift+\ = toggle monitoring of live regions They look a little bit like a patchwork quilt to me, but I'll defer to Mike's UI design on the choices. I have a comment in addition to the Shift+y one discussed above. In the way they are used here, "previous" and "next" refer to navigational location on the page, which makes sense. The word "last" threw me, however, and I went looking for "previous". I then realized "last" was in the temporal sense of the word. So, when I look at this help text for the user: + _("Goes to previous live region.")) + _("Goes to next live region.")) + _("Goes to last live region.")) I wonder if a different word choice can be made for "last" (e.g., "current")? Or, do we just need to live with the fact that we are dealing with words that can describe both relative aspects of time and location?
> I think we need Mike's input on this. It seems as though these messages may be > a bit like instant messages, so maybe we should consider providing a user > experience similar to what we get with gaim/pidgin (Orca+#). But, that seems > to conflict with bookmark stuff. In the Orca team meeting, we decided to shuffle the bookmarks keys around and use the Orca+# keys for things with the flavor of "read last nth message". See bug 500193 for more information.
Created attachment 100139 [details] [review] thirteenth version of live regions Added Orca+Fn (n = 1-9) key bindings for reviewing previous messages.
As long as this works for Mike and meets his UI approval, it looks good to me. :-) Thanks! Thirteen is my lucky number.
I tested this one last night and it seemed to be working.
Created attachment 100318 [details] [review] fourteenth version of live regions This version has the following changes: 1) idle_add is only called (or return True from callback) if the queue is not empty. This increased performance by ~25%. Thanks Rob! 2) Changes to increase pylint score. Still having problems with pyatspi.RELATION*. 3) Basic cleanup not related to pylint, mainly whitespace.
Created attachment 100392 [details] [review] fifteenth version of live regions Changes include the following: 1) moved embed test for text-changed events from Gecko.onTextInserted() to liveregions._getUtterances(). This work include minimizing calls to IText.getText(). 2) fixed isAriaWidget() return value (included parenthesis).
Pylints to a 10.00 for all the files (yeah!) Runs with no stack traces (yeah!) New strings marked for translation have comments for translators (yeah!) New file added to POTFILES.in and the docs/pydoc/Makefile.am (yeah!) Seems to do what it is supposed to do (yeah!) Commit! Thanks for the work! :-)
committed to trunk. Marked as fixed.
Added test for LIVE_OFF in handleEvent(). Committed to trunk.