:
> 
> 1)  We will need to be able to track when a page has finished loaded and act on
> events that occur only after the page is finished loading.  We do not want to
> act on children-changed events that occur during a page load.
> 
This can be tracked with Gecko._loadingDocumentContent .  This variable is False when the page is not being loaded.

> 2)  We will need to address object labeling issues.  Context of a live region
> change will be very difficult to determine without proper labeling.  Hopefully
> the method that is in place will be satisfactory for this purpose.
> 

Unfortunately, Gecko.getDisplayedLabel() does not return the label for many real life live regions.  What I am seeing quite often is the use of tables without the use of <th> elements (these elements will form a relation).  I may need to implement a live region specific getDisplayedLabel() that is specific for tables where the first column is assumed to be the label.  Can we always assume the first column is a label?  How about the first row?  This are questions I currently don't have an answer for.  

(In reply to comment #2)
> Unfortunately, Gecko.getDisplayedLabel() does not return the label for many
> real life live regions.  What I am seeing quite often is the use of tables
> without the use of <th> elements (these elements will form a relation).  I may
> need to implement a live region specific getDisplayedLabel() that is specific
> for tables where the first column is assumed to be the label.  Can we always
> assume the first column is a label?  How about the first row?  This are
> questions I currently don't have an answer for.  

I'm not sure we can assume this.  :-(  Would the label for/by semantics that we see in regular HTML apply, however?  If so, we might try to influence the ARIA/AJAX toolkit providers to provide more best practices examples that use the label for/by stuff.

> I'm not sure we can assume this.  :-(  Would the label for/by semantics that we
> see in regular HTML apply, however?  If so, we might try to influence the
> ARIA/AJAX toolkit providers to provide more best practices examples that use
> the label for/by stuff.
> 

Labeling works just fine when best practices are followed.  I am trying to enhance the live region experience when there is no markup at all.  

Created attachment 102012 [details] [review]
first version of Accommodate no ARIA markup for live regions

This is the first version of live region support for pages that do not have any ARIA markup.  Here is a list of features that are provided:

1) Enhanced labeling detection for table based live events.  Works for all real life sites encountered (finance.google.com, finance.yahoo.com, live scores at CBS sports ...)

2) Message filtering.  Messages now consist of a 'label' and 'content' list of utterances.  Filtering begins by comparing the label of the first message to the label of all queued messages.  If the labels are the same, the contents of the newer message is appended to the contents of the first message and the newer message is deleted.  Duplicate content utterances that may arise from the first are then removed.  This seems to drastically reduce verbosity while still retaining context.
3) Increased a message time to live to 45 seconds.

speech.isSpeaking() is a key method call for live regions because there needs to be coordination between what is being announced and the object that is generating the announcement.  This coordination is most important for goLastLiveRegion() where a user hears an announcement and wants to go to that live region.  However, what seems to be happening is that isSpeaking() is returning False at times when messages are being output.  At first I thought this was a problem in gnomespeechfactory or lower.  It very well may still be.  However, the problem goes away if I put a small sleep in pumpMessages() (the event callback) for cases when isSpeaking() returns True.  This leads me to think that 1) isSpeaking() is not thread safe.  Will, is this possible? or 2) the large number of callbacks increases my odds of finding a hole in isSpeaking().  The sleep reduces the number of callbacks so I just don't see it in this case.

Adding the sleep as described in the previous comment causes the key events not to be triggered.  Something strange is happening with respect to the event handler.  Will, can you double check how I am doing the idle_add().  The logic is split between pumpMessages() and handleEvent().  It doesn't seem to work any other way.  I have tried the following in handleEvent() with no luck:

if message:
    self.msg_queue.enqueue(message, politeness, event.source)
    if len(self.queue) == 0:
            gobject.idle_add(self.pumpMessages)

I have also tried always returning False in pumpMessages().  No luck either. 

(In reply to comment #6)
> live region.  However, what seems to be happening is that isSpeaking() is
> returning False at times when messages are being output.  At first I thought
> this was a problem in gnomespeechfactory or lower.  It very well may still be. 
> However, the problem goes away if I put a small sleep in pumpMessages() (the
> event callback) for cases when isSpeaking() returns True.  This leads me to
> think that 1) isSpeaking() is not thread safe.  Will, is this possible? 

It may be possible.  In gnomespeechfactory.py, the only thing maintaining state about isSpeaking is the __isSpeaking field which is set via calls to __speak and stop.  I think the speech progress callback will also set it (see the notify() method around line 550).

> or 2)
> the large number of callbacks increases my odds of finding a hole in
> isSpeaking().  The sleep reduces the number of callbacks so I just don't see it
> in this case.

This might be the case, too.  I think the isSpeaking stuff is always going to be a bit error prone, though.  If I recall correctly, it was originally added for FireVox, expressly for live region support.

With respect to handleEvent and pumpMessages, you might be on to something.  This part of handleEvent should probably be moved to the end of the method (changing "if len(self.msg_queue) == 0" to "if len(self.msg_queue) > 0"):

        # Add a callback if the queue is empty because we are about to add 
        # something to it.
        if len(self.msg_queue) == 0:
            gobject.idle_add(self.pumpMessages)

The reason being is that, depending upon which thread is calling handleEvent, pumpMessages might be called before something is added to the queue.  In looking at the code, though, it seems as though handleEvent is only called from methods that get called from the gidle thread, so I'm not sure what's going on.  You could always put some error condition checking at the very top of pumpMessages to see if it is ever called when the msg_queue is empty -- if that should never be the case, but we run into it, then something might awry somewhere.

Created attachment 102344 [details] [review]
second version of Accommodate no ARIA markup for live regions

This version is looking very good.  Here is the major change:

Live regions that do not have an id (derived from HTML markup) new use the path to the doc frame for persistence of that object's politeness.  At first, using the object's path seemed like it would be too expensive but it turned out not to be and worked into the bookmark scheme quite well.  Now when a live region object containing no ARIA markup is seen, it is automatically registered into the politeness override dictionary.  These registered objects (actually their paths) can later be examined in the goTo[Next|Prev]LiveRegion() methods to help find objects without markup.  The end result is that the user now has the ability to set politeness on any object, go to any object that is marked as a live region or has registered as a live region and to change the politeness of that object and can persist this setting via the bookmarks functionality.

Try this:

1. Go to http://bugzilla.gnome.org/query.cgi?format=advanced
2. Tab to the classification list
3. Down arrow

This will cause the lists to the right to repopulate (adding children).  Orca starts speaking the new additions.

That opens up a host of issues and illustrates how hard it is to filter these events and correctly classify them as a live region event.  I don't see any useful object attributes we could use for this example either.  Were we suppose to see an 'input_from_user' object attribute?  I don't see it so I am stumped at the moment.  Hopefully, we will not have to resort to a global on/off toggle for live regions.

Aaron,
How can I filter the object:children-changed and object:text-changed events in Joanie's example as described in comment #10?  They are currently being interpreted as a live region which is no good.  The following is some debug showing:

for object:children-changed events
event.type, event.source, event.any_data, event.source.getAttributes(), event.any_data.getAttributes()

for object:text-changed events
event.type, event.source, event.source.getAttributes()

object:children-changed:add:system [list | pessulus sabayon] [list item | pessul
us] ['tag:SELECT', 'id:product'] ['formatting:block', 'posinset:1', 'tag:OPTION'
, 'setsize:2']
object:children-changed:add:system [list | pessulus sabayon] [list item | sabayo
n] ['tag:SELECT', 'id:product'] ['formatting:block', 'posinset:2', 'tag:OPTION',
 'setsize:2']
object:children-changed:add:system [list | docs general] [list item | docs] ['ta
g:SELECT', 'id:component'] ['formatting:block', 'posinset:1', 'tag:OPTION', 'set
size:2']
object:children-changed:add:system [list | docs general] [list item | general] [
'tag:SELECT', 'id:component'] ['formatting:block', 'posinset:2', 'tag:OPTION', '
setsize:2']
object:children-changed:add:system [list | 0.1 0.12 0.15 0.16 0.17 0.18 0.2 0.3 
0.4 0.5 0.6 0.7 0.8 0.9 0.9.x 0.10.x 2.11.x 2.12.x 2.15.x 2.16.x 2.17.x 2.18.x 2
.19.x 2.20.x SVN trunk unspecified] [list item | 0.1] ['tag:SELECT', 'id:version
'] ['formatting:block', 'posinset:1', 'tag:OPTION', 'setsize:26']
object:children-changed:add:system [list | 0.1 0.12 0.15 0.16 0.17 0.18 0.2 0.3 
0.4 0.5 0.6 0.7 0.8 0.9 0.9.x 0.10.x 2.11.x 2.12.x 2.15.x 2.16.x 2.17.x 2.18.x 2
.19.x 2.20.x SVN trunk unspecified] [list item | 0.12] ['tag:SELECT', 'id:versio
n'] ['formatting:block', 'posinset:2', 'tag:OPTION', 'setsize:26']

Here is the description of the event-from-input object attribute that I mentioned in the previous comment.  http://developer.mozilla.org/en/docs/AJAX:WAI_ARIA_Live_Regions/API_Support#Determining_if_event_was_from_user_input

(In reply to comment #11)
> Were we suppose
> to see an 'input_from_user' object attribute?  I don't see it so I am stumped
> at the moment.

That's what :system is. If an event is *not* from the user it's tagged with ":system" on the end.

We had to do it that way because of ATK/AT-SPI's events being asynchronous.

In IRC, Aaron indicated that without ARIA markup there is not enough information in an event to determine if it is a live region event. :system will not work for Joanie's example because it is technically a user driven change but is Javascript driven and thus the :system event.type appendage.  

Aaron indicated that filtering these events will have done through heuristics.  Here is what I am currently using to filter these events.

1) Is it a :system event?
2) Is the event.source within the document frame?
3) Is the page loading complete?

I could also include:

1) Is the event.source *not* within a form?

Any others?  Another solution of course is to provide a global on/off toggle for live regions.

Adding the filter of "is it an element within a form" may be too generic. You may want some things such as labels etc. to be spoken (for example a character counter that goes down while you type into a textbox). So, why not settle on a set of form controls such as "option" that you'd probably never associate with live regions? Repopulating comboboxes is something I would not consider something that should get spoken in whole because the selection is changing within one combobox. So Joanie's example would be excluding speaking of "option" elements if they update via JS.

Created attachment 102527 [details] [review]
third version Accommodate no ARIA markup for live regions

The patch includes the following changes from the previous patch.

1) Gecko.handleAsLiveRegion() was created to serve as our live region filter.  The new filter includes looking for object attribute 'tag':SELECT and 'tag':OPTION.  This has fixed the comment #10 example, but I am not sure if this filter is complete enough.  Hopefully this will filter out all events triggered by dynamically created comboboxes.  Should we be aware of any other dynamically created widgets that we can filter out?  

Also of note is the impact on performance this method can have.  It is small now and needs to remain that way.  A good test case to study the impact on performance is the example in comment #10. Arrowing down from 'Deprecated' to 'Desktop' causes a very large number of events.  I estimate that the addition of handleAsLiveRegion() causes about a 15% increase in time between list item announcements.  Really not too bad for this rare and extreme case.

2) I removed the alert/tooltip support for now.  It caused a minor performance hit and needed work anyway.  I should probably reopen bug #462883 or create a new one,

Created attachment 102601 [details] [review]
fourth version of Accommodate no ARIA markup for live regions

The example in comment #10 has literally hundreds of events.  The performance impact is too large to do a getAttributes() call on all events.  To address this I added the key command Orca+Shift+\.  I saw two routes I could take.  

1)  I could add settings.inferLiveRegions and use it exclusively to toggle whether I do the expensive getAttributes() call.  

2) I could replace Gecko.liveRegionsOn with settings.inferLiveRegions and attach the new key binding to toggling settings.inferLiveRegions.  The toggle essentially turns on|off live support with the default being off.

I wrestled with the two and decided on choice two because it completely disables live region support instead of partially disabling it (text-changed events are still going through).  I feel that partially disabling the feature might add some user confusion or the user unwittingly accepting the live region experience as complete when it is actually partially disabled.

In addition to the new key command, I also changed some method names to more accurately reflect their functionality.

> I wrestled with the two and decided on choice two because it completely
> disables live region support instead of partially disabling it (text-changed
> events are still going through).  I feel that partially disabling the feature
> might add some user confusion or the user unwittingly accepting the live region
> experience as complete when it is actually partially disabled.

I'll let Mike comment on the user experience portion.  If he's OK with it, I am. 
A question I have, however, is if we even need Orca+Shift+\ if choice two is used?  That is, perhaps the existing Shift+\ binding could be used?

(In reply to comment #18)

> A question I have, however, is if we even need Orca+Shift+\ if choice two is
> used?  That is, perhaps the existing Shift+\ binding could be used?
> 

No, I don't think that will work.  Shift+\ turns all the politeness levels to LIVE_OFF.  The user can then navigate to a live region and advance the politeness level for a single live region.  We want that single live region to go through so Shift+\ cannot serve as a global on|off.  In essence, Shift+\ is used when the user wants live regions to shut up for now but is still interested in monitoring live regions, including registering LIVE_NONE events.

I'm OK with Scott's changes here.  It seems as though we have no other way to work around the related performance hit.  Just to confirm:  Live regions will now be turned off by default.  
I think I'd also like to see a "monitor live regions" checkbox added to the firefox tab of the orca prefs. 

Even live regions marked with ARIA markup will be off by default? Or just the unmarked ones?

(In reply to comment #21)
> Even live regions marked with ARIA markup will be off by default? Or just the
> unmarked ones?
> 
As it stands now, all live regions will be off by default.  The process of filtering using the object attributes is very costly when looking at the example in comment #10.  I'm open to any idea where the filtering can be done in a performant manner.  The long term goal here is to bring in per URL scripting that will open and close this filter to live region monitoring.

I'm against needing a per-URL script to make live regions work. 

How about using collection when a page loads or at the first live change to see if there are any live regions? If no, turn off support for the page by default. If yes, turn it on. This way the processing is done up front.

On Windows this problem doesn't exist since the ATs can listen to events in process. Long term we may need infrastructure changes to provide something like that capability on Linux. At least the in-process part of the infrastructure could do some more advanced kind of event filtering based on common needs.

> I'm against needing a per-URL script to make live regions work. 
> 
I see your point but don't discount this idea so fast.  It would satisfy our performance issues, thus providing the best overall user experience.  Turning live regions on is just a key command away.  

> How about using collection when a page loads or at the first live change to see
> if there are any live regions? If no, turn off support for the page by default.
> If yes, turn it on. This way the processing is done up front.
> 
This is a good idea for pages that have proper markup, but not much good for pages that have no markup.  The Collection test would have to be on a page by page basis.  This might increase page loading times for us.  I will take this into consideration though.

> On Windows this problem doesn't exist since the ATs can listen to events in
> process. 
Yes, we are certainly feeling the pain of IPC.

> Long term we may need infrastructure changes to provide something like
> that capability on Linux. At least the in-process part of the infrastructure
> could do some more advanced kind of event filtering based on common needs.
We would see a very nice performance increase if we could get this done.


The main question to ask is how often will the user come across pages like those in comment #10 and is the 15% ding in performance on this page a show stopper?  If it is, I only see minor tweaks we can do.  If it is not, we might revert back one patch.  

I'll give it all some thought while I await the required GUI fix.

My main objection is if the we need a script to turn live regions on when the author already indicated there are live regions.

I think it would be useful to collect that on page load for the page summary anyway. It's very useful info for a user to hear there are 2 live regions on the page they just loaded.

In general I don't like it when the limitations of the Linux accessibility infrastructure get in the way of a good solution for end-users. Per-URL scripts should not be necessary for most of the web to "just work".

(In reply to comment #26)
> My main objection is if the we need a script to turn live regions on when the
> author already indicated there are live regions.

If my understanding of this problem is correct, we would not be having this discussion if content providers were doing the right thing.  :-(  Instead, this bug is to accommodate the situation where there is not proper ARIA markup for live regions.  As such, this bug attempts to infer that something is a live region.  If I'm wrong about this understanding, then the rest of this e-mail may not make sense.

One of the other alternatives we discussed in comment #17 was to not disable live region support altogether, but instead to just disable the inference code that attempts to detect live regions that do not have ARIA markup.  That alternative would work as follows:

  1) Live region support is enabled by default, but the inference code is disabled by default.  As such, Orca users will not need to suffer a performance hit for every page they visit just because a subset of content providers do the wrong thing.  NOTE that we should also proactively lobby the recalcitrant offenders to get them to do the right thing in addition to the reactive approach of trying to work around their failures.

  2) We can provide a keystroke to toggle the inference code and a user-settable customization to have it enabled by default.  So, as users go to pages they *know* are doing some sort of broken live region support, they can force the inference code to do the right thing.  I suspect that most users will be in this boat (i.e., I'm going to check the latest scores on my favorite web page, I'm going to check stocks on my favorite finance page, etc.).

  3) As we do per-URL scripting, we could potentially provide scripts to turn on the inference code for common pages whose content providers do the wrong thing.  Ideally the script would send a bug report to the content provider, too.  :-)

A third alternative would be to just not do any inference code and rely upon users to activate the greasemonkey script referenced in the opening comment.

> I think it would be useful to collect that on page load for the page summary
> anyway. It's very useful info for a user to hear there are 2 live regions on
> the page they just loaded.

This option would introduce a delay in processing every page, even when collections are used.  We could, however, definitely open this idea up for discussion with end users on the Orca list.  If there is strong interest from the user community in automatically getting page summary information for every page they visit, we can open this as a separate RFE.

Thanks!

> If my understanding of this problem is correct, we would not be having this
> discussion if content providers were doing the right thing.  :-(  Instead, this
> bug is to accommodate the situation where there is not proper ARIA markup for
> live regions.  As such, this bug attempts to infer that something is a live
> region.  If I'm wrong about this understanding, then the rest of this e-mail
> may not make sense.

Will, your understanding is not quite at 100%.  The code that infers a live region (the patch) and the code that identifies proper markup (in trunk) are the same with respect to performance.  Both approaches require a single getAttributes() call.  The first approach looks for the 'container-live' attribute and the other approach looks for the tag:SELECT and tag:OPTION to filter out dynamically built comboboxes.  So, with respect to the key command toggle, it needs to be all off or all on.  We can't just turn on|off the inference code without the problems I described in comment #17.

What spurred this conversation is the Bugzilla example which illustrates a small performance issue.  Note, this performance problem is in trunk too.  I consider this performance issue to be somewhat significant not because this problem will occur on a number of pages.  I really think this is very rare.  My thinking was it was significant because it was Bugzilla.  I would hate for our sight impaired developers to suffer on this page.

After all this discussion I think our approach should be the following:

1) Use the inference code that is shown in the latest patch.

2) Use Orca+Shift+\ to completely disable|enable live regions.  Part of this process needs to include flushing the live region queue when live regions are disabled.  This is current not implemented.

3) Enable live regions by default.  This will allow live regions to be announced if they are available and will not introduce any more performance problems than are already in trunk.  The key command toggle will actually give a savvy Orca user a performance boost (compared to trunk). 

4) This scheme will feed nicely into per URL scripts because the enabling|disabling of live regions would be controlled by a single variable.


Still needing to be implemented:

1) live region queue flushing when live regions are disabled.
2) I will also need to flush the politeness override dictionary after page loads because this data structure will continue to grow as LIVE_NONE events are seen.  Will, what is the best way to handle this from Gecko?  Simply make a call to the live region manager from Gecko.onDocumentLoadComplete() ?

> Will, your understanding is not quite at 100%. 

Thanks for the clarification!

> After all this discussion I think our approach should be the following:
> 1) Use the inference code that is shown in the latest patch.
> 2) Use Orca+Shift+\ to completely disable|enable live regions.  Part of this
> process needs to include flushing the live region queue when live regions are
> disabled.  This is current not implemented.
> 3) Enable live regions by default.  This will allow live regions to be
> announced if they are available and will not introduce any more performance
> problems than are already in trunk.  The key command toggle will actually give
> a savvy Orca user a performance boost (compared to trunk). 
> 4) This scheme will feed nicely into per URL scripts because the
> enabling|disabling of live regions would be controlled by a single variable.

Sounds like a good thing.  This will also give Bugzilla users the chance to toggle to the feature to see what the performance difference is really like.

> 2) I will also need to flush the politeness override dictionary after page
> loads because this data structure will continue to grow as LIVE_NONE events are
> seen.  Will, what is the best way to handle this from Gecko?  Simply make a
> call to the live region manager from Gecko.onDocumentLoadComplete() ?

I think this might do it, though you may also need to do it in onDocumentLoadStopped.  What would the implication be if you did this when the document loading started?  

BTW, if one loads a page with live regions in one tab, and another page with live regions in another tab, where in the code is that handled?

> > 2) I will also need to flush the politeness override dictionary after page
> > loads because this data structure will continue to grow as LIVE_NONE events are
> > seen.  Will, what is the best way to handle this from Gecko?  Simply make a
> > call to the live region manager from Gecko.onDocumentLoadComplete() ?
> 
> I think this might do it, though you may also need to do it in
> onDocumentLoadStopped.  What would the implication be if you did this when the
> document loading started?  

I just need a valid getURI() call.  Can I be assured this will be valid at this time?  I will have to experiment to find out.

> BTW, if one loads a page with live regions in one tab, and another page with
> live regions in another tab, where in the code is that handled?

This is an excellent question and something I have failed to address.  Is this a Firefox bug?  Aaron indicated it was when I brought this up several months ago.  However, is it something we could exploit and possibly provide a feature for?  Would a user want to hear live regions from another tab?  I just don't know.  At the moment all live regions, regardless of tab, are treated equal.

I overlooked the inDocumentContent() call in handleAsLiveRegion().  This is certainly an expensive call and is not present in the trunk version.  We will take a performance hit for all live region events and this contributes quite a bit to the performance hit we see on Bugzilla.  

This code is in place to filter out events from the toolbars and menubar.  I will experiment on a more performant way of dealing with these events.

Here is a sampling of events derived from user events with the toolbars/menubars:

From arrowing in URL bar
object:children-changed:add [autocomplete | Location] ['haspopup:true', 'tag:textbox', 'id:urlbar']
object:text-changed:insert [entry | Location] ['haspopup:true', 'tag:html:input']

from typing in Google search
object:text-changed:insert [entry | Search Google] ['event-from-input:true', 'haspopup:true', 'line-number:1', 'tag:html:input']
object:children-changed:add [menu | ] ['event-from-input:true', 'tag:panel', 'id:PopupAutoComplete']
object:text-changed:insert [entry | Search Google] ['haspopup:true', 'tag:html:input']

From menu.  Note: we don't get an event from every menu opening and when we do it is an explosion of the same event (about 10 of the same event) 
object:children-changed:add:system [menu | History] ['haspopup:true', 'tag:menu', 'id:history-menu']

Created attachment 102843 [details] [review]
fifth version of Accommodate no ARIA markup for live regions

This version eliminates the expensive inDocumentContent() call in handleAsLiveRegion() with a set of heuristics in an attempt to minimize IPC.  It seems to do a good job filtering with only minimal impact on performace.  My only question is in regards to the object:text-inserted part of handleAsLiveRegion().  Is 'xul:description' and 'xul:label' too generic, thus creating a too restrictive filter?

I also addressed the flushing of the politeness override data structure by making a live region manager call from onDocumentLoaded().

I am also now flushing the message queue when the user turns off live region support.  Should I also flush the queue when all politenesses are turned off?

Will, could you also respond to my answer to an earlier question of yours.
> BTW, if one loads a page with live regions in one tab, and another page with
> live regions in another tab, where in the code is that handled?

This is an excellent question and something I have failed to address.  Is this
a Firefox bug?  Aaron indicated it was when I brought this up several months
ago.  However, is it something we could exploit and possibly provide a feature
for?  Would a user want to hear live regions from another tab?  I just don't
know.  At the moment all live regions, regardless of tab, are treated equal.

>  Would a user want to hear live regions from another tab?  I just don't
> know.  At the moment all live regions, regardless of tab, are treated equal.
> I'd have to say the answer to this question is a resounding no.  

From what I understand, chat is a good example of where one would want to hear things from another tab. Basically if someone writes your name in a line or something from a list of keywords you're interested in, you may want to hear it. In that case scripting for chat apps or per-URL scripting would be useful.

Or for a more general solution it could be based on politeness. Perhaps the user would like to hear assertive or rude events from another tab. After all, Chatzilla plans to mark "important" chat messages as assertive.

One thing is for sure, if we change Firefox not to fire any events in other tabs, it would make any of these scenarios impossible to deal with.

Understood.  I guess orca would need a way to determine if the live region was coming from the tab with focus.  If we are going to speak messages from other tabs this functionality should be off by default.  

Aaron, Marco:  Would you please review my heuristic that identifies an event/object as a live region.  The goal here is to limit IPC as much as possible and still retain a good filter.  The following is simplified python with lots of comments.  Let me know if you don't understand the code.

def handleAsLiveRegion(event):
        """Returns True if the given event (object:children-changed, object:
        text-insert only) should be considered a live region event"""

        # We will try to eliminate objects that cannot be considered live
        # regions.  We will handle everything else as a live region.  We
        # will do the cheap tests first
        if self._loadingDocumentContent \  # Is document being loaded?
              or not  settings.inferLiveRegions \  # Is our global toggle on?
              or not event.type.endswith(':system'): # Is it a system generated event?
            return False

        # Ideally, we would like to do a inDocumentContent() call to filter out
        # events that are not in the document.  Unfortunately, this is an 
        # expensive call.  Instead we will do some heuristics to filter out
        # chrome events with the least amount of IPC as possible.

        # event.type specific checks
        if event.type.startswith('object:children-changed'):
            # This will filter out lists that are not of interest
            stateset = event.source.getState()
            if stateset.contains(pyatspi.STATE_FOCUSABLE) \
                   or stateset.contains(pyatspi.STATE_FOCUSED):
                return False
            # Now we need to look at the object attributes
            attrs = self._getAttrDictionary(event.source)
            # This filters out menu events
            if attrs.has_key('haspopup') and attrs['haspopup'] == 'true':
                return False
            # We see this one with the URL bar opening (sometimes)
            if attrs.has_key('tag') and attrs['tag'] == 'xul:richlistbox':
                return False

        else: # object:text-inserted event
            attrs = self._getAttrDictionary(event.source)
            if attrs .has_key('tag'):
                # This might be too restrictive but we need it to filter
                # out URLs that are displayed when the location list opens.
                if attrs['tag'] == 'xul:description' \
                                or attrs['tag'] == 'xul:label':
                    return False

        # It sure looks like a live region
        return True

Created attachment 102933 [details] [review]
sixth version of Accommodate no ARIA markup for live regions

This version addresses the following:

1) minor translator string changes.
2) live region events that are on a hidden tab are no longer handled as live region events.  This was handled by looking for state VISIBLE.  This surprised me because I though objects that were scrolled out of view were considered not VISIBLE, but this is not the case.  Is this a Firefox bug or something I can exploit?  It would be great if it is something we can exploit because this additional logic would almost come for free.

I also checked out the workaround where the user would open up a separate window to monitor a live region containing page they were interested in.  I believe this is a viable solution because the live region updates are announced from the second window and the user has the ability to navigate a page on the other window.  I would like Mike's opinion on this as well.

(In reply to comment #38)
> 1) minor translator string changes.

Thanks.  :-) 

> 2) live region events that are on a hidden tab are no longer handled as live
> region events.  This was handled by looking for state VISIBLE.  This surprised
> me because I though objects that were scrolled out of view were considered not
> VISIBLE, but this is not the case.  Is this a Firefox bug or something I can
> exploit?  It would be great if it is something we can exploit because this
> additional logic would almost come for free.

I think we can exploit it.  State SHOWING is used to indicate that an object is actually meant to be painted.  I remember it with a stupid, doesn't really work, mnemonic: SHOWING burns PHOSPHOROUS.  You see, there's an S and an H in both.  It's the only way I remember.

So, in the team meeting today, we agreed Mike would test this and give his feedback.  If he's OK with it, I say commit and thanks for persevering on this bug.  :-)

Oh yeah - we also need a quick sanity check from Aaron and Marco on the heuristics in comment #37.

To me, this looks OK! It filters out everything we don#t want: Menus, listboxes/comboboxes updating, and stuff from the URL bar.

Here are my findings from an informal performance analysis.

Test Case
I used the bugzilla example linked in comment #10 and focused exclusively on arrowing down from the 'Deprecated' list item to the 'Desktop' list item.  There are 1128 object:children-changed events triggered with this single user command.

Approximately 10 seconds for Firefox to change focus to 'Desktop' list item.  We do not get our first event until focus has changed.

The following is a list of time taken to perform additional processing by Orca.
no Gecko object:children-changed listener  - 20 secs. 
return immediately in onChildrenChanged()  - 23 secs.
return immediately in handleAsLiveRegion() - 23.2 secs.
return after 'cheap' tests                 - 23.8 secs.
return after 'cheap' + getAttribute()      - 24.4 secs.
return after 'cheap' + getState()          - 24.6 secs.
return after complete processing           - 25.1 secs.
Note: complete processing is defined as 'cheap' + event.type string compare + getState() + contains() operation.  The total time before a user hears the announcement is approximately 35 secs (Firefox time + Orca time)

Although this is an informal performance analysis, several conclusions can be made.
1) 1128 object:children-changed events is an extremely rare case.  
2) Approximately 30% of the total time is due to Firefox.
3) Approximately 57% of the total time is due to unrelated Orca processing.
4) Approximately 8% of the total time is due to handling object:children-changed.
5) Approximately 6% of the total time is due to filtering these type of events and classifying them as a live region.

Interesting observations:

1)  The tabpanel example seen here http://archive.dojotoolkit.org/nightly/dojotoolkit/dijit/tests/layout/test_TabContainer.html adds content to a scroll pane when moving from 'Tab2' to 'Tab3' in the first tab panel.  This triggers both text-inserted and children-changed events on the scroll page.  The problem is that this object's states cannot be used to filter it out because it is not FOCUSED, FOCUSABLE and is VISIBLE.  For this example I could use the object attribute 'xml-roles' != 'container-live', but I can foresee similar cases when ARIA markup is not used.  The xml-roles check is probably a good one, but what do we use to filter out scroll panes without markup.  I could start using role name but is this the best generic solution?  most performant?

2) The tab panel seen here http://test.cita.uiuc.edu/aria/tabpanel/view_html.php?title=Tab%20Panel%20Example%201&ginc=includes/tabpanel1_xhtml.inc&gcss=css/tabpanel1_xhtml.css&gjs=../js/globals.js,../js/widgets2_xhtml.js,js/tabpanel1_xhtml.js has an extra section that has not been removed from the AT-SPI hierarchy using role="presentation".  This causes a single text-inserted and a single children-changed event to be triggered on this object during a page reload *after* the load complete event.  This event is very hard to filter with our heuristic.  The result is 'crust page' is spoken during the page reload.  What is interesting is that this problem goes away when role="presentation" is added to this object.  This begs the question: how many poorly written ARIA widgets (possibly well written ones too) are going to have similar problems when they build dynamically?

(In reply to comment #42)
> Here are my findings from an informal performance analysis.
> 
> Test Case
> I used the bugzilla example linked in comment #10 and focused exclusively on
> arrowing down from the 'Deprecated' list item to the 'Desktop' list item. 

I repeated this test case with two versions of Orca: trunk (svnversion 3469) and trunk with the 6th version of the patch (http://bugzilla.gnome.org/attachment.cgi?id=102933) applied.

The main test I did was:

1) Put focus on the 'Deprecated' list item and everything was in steady state
   (e.g., Orca wasn't processing any events)
2) Wait until my wall clock second hand hit 0 seconds.
3) Arrow down and then press tab.
4) Wait for Orca to speak and look at my wall clock.

The four test cases I did were:

1) No Orca running at all -- average time for things to settle down was around 20-25 seconds

2) Orca from trunk running (svnversion 3469).  The average time between the time I pressed the down arrow and heard Orca speak was around 50-55 seconds.

3) Orca from trunk with the patch applied.  The average time was identical to Orca from trunk.

4) Orca from trunk with the patch applied, but with monitoring live regions turned off via Insert+Shift+\.  Remarkably, and unexpectedly, the average time went up a couple seconds.  Yes, I'm sure I turned monitoring *off*. I even reran the test with "orca.settings.inferLiveRegions = False" in my ~/.orca/user-settings.py.  

So, from my "from the user's perspective" testing, I'm not sure I'm seeing the patch introducing any performance degradation on the test case.  Mike, if you have other test cases where the degradation is measurable, please post them.

In any case, given the new cases from comment #43, I'm beginning to wonder how tractable this problem is going to be and/or if something might need to do some filtering or role adjustment before things get to Orca?  For example, can it be possible to write a greasemonkey script to fix these kinds of things up, or is that kind of thing outside the domain of what the incredible monkey can do?

A side note to comment #43:

The UIUC link is broken in comment #43.  It may have been updated.  Instead, try

http://test.cita.uiuc.edu/aria/tabpanel/view_inline.php?title=Tab%20Panel%20Example%201&ginc=includes/tabpanel1_inline.inc&gcss=css/tabpanel1_inline.css&gjs=../js/globals.js,../js/widgets_inline.js,js/tabpanel1_inline.js

The scrollpane that is giving me problems implements the Text interface.  However, all other scrollpanes that I have seen do not implement the Text interface.

Created attachment 103564 [details] [review]
seventh version of live regions

This following was addressed in this version:

1) the live region filter (handleAsLiveRegion()) was further tweaked.

It should be noted that speech.isSpeaking() performs quite differently on different speech services.  For instance, isSpeaking() almost always (if not always) returns False when using speech dispatcher but is much more accurate when using GNOME speech.  This is very important for live region support for two reasons:  1) the object associated with the last output message is cached and is used for the "go to last live region" user function.  If isSpeaking() is not working properly, the announced message will not be in sync with the cached object.  2) Filtering of utterances is done on the queued messages.  There is no way to filter these messages if the message has already been sent to the speech synthesizer as is the case when isSpeaking() always returns False (or is "off the mark").

One thing I'm noticing is that speech interupt seems quite sluggish.  For example: If you go to the finance sites at either google or yahoo and start listening to the quotes as they appear pressing CTRL takes about a half second to stop speech.  I'm not seeing any other real performance hits related to this patch so far.     

Will,

Here's my take on what Mike is experiencing (see comment #47).  When CTRL is hit, speech.stop() is called which flushes the speech synthesizers queue.  isSpeaking() then returns False so the live region manager instantly calls speech.speakUtterances() and is able to sneak the message through before other output is queued (perhaps from tabbing).  Mike may argue that the current message that is being output takes too long to stop, but I don't see much difference between a live region message and a message derived from a tab event.  The time to stop it via speech.stop() is the same.  Here are some suggestions I have that may help the user experience.

1) liveregions.pumpMessages() could also check for the last keyboard event.  If the event was long enough in the past then I would queue a message.  This seems to help navigating around the finance pages, but this strategy fails when we get a message that is longer (in time) than our defined time difference.

2) We could flush the live regions queue on a keyboard event.  The problem here is lost information.

Any suggestions?

Mike, which speech engine are you using?  I've seen slow response time from Cepstral when it comes to interrupting speech.

(In reply to comment #49)
> Mike, which speech engine are you using?  I've seen slow response time from
> Cepstral when it comes to interrupting speech.
> 
espeak with gnome-speech 

Another reason we are seeing a delay is due to Firefox's slow response to a tab command when live region support is on.  I am seeing up to a half second delay from the time the tab button is pressed to changing focus in Firefox.  This time delay is only milliseconds when live region support is not on.  This may indicate that the additional computational power needed to support live regions (live region support itself, additional speech output) is the root cause of the problem.

(In reply to comment #51)
> Another reason we are seeing a delay is due to Firefox's slow response to a tab
> command when live region support is on.  I am seeing up to a half second delay
> from the time the tab button is pressed to changing focus in Firefox.  This
> time delay is only milliseconds when live region support is not on.  This may
> indicate that the additional computational power needed to support live regions
> (live region support itself, additional speech output) is the root cause of the
> problem.

This sounds like a reasonable possibility.  In IRC, I asked Scott to run a test to see if key presses events were not getting to Orca quickly when live regions were on, and the answer was yes.  Since we depend upon key press events to interrupt speech right now, that's probably what's causing the delay.  In other words, this problem is most likely caused by latency introduced by FF vs. Orca.

So, Mike and Scott - if you're comfortable with this patch modulo the performance issue, I'd like to see it checked in and a separate bug opened for the performance concerns.  Are you OK with that?

Scott - make sure also to send out a string change announcement once you check this patch in.

After seeing the results of Scott's testing and talking with Will I think checking this in is a good idea.  Perhaps we should still leave it "testing required" for a while though.  

(In reply to comment #53)
> After seeing the results of Scott's testing and talking with Will I think
> checking this in is a good idea.  Perhaps we should still leave it "testing
> required" for a while though.  
> 

Just a friendly reminder: if you check it in, it should be marked as [pending] since we use [testing required] as an indicator we want people to test something before we check it in.

committed to trunk.  Bug to remain open for further testing.

Scott, this seems to work well but what do we want to do about the extra chatter on the gmail login screen?  If you want to make that a separate bug I'll verify this one. 

Mike, the gmail login screen helped me to identify an 'atomic' issue so I am grateful that you pointed it out.  I committed that fix last week as part of the alert/tooltip bug.  Now what you hear on gmail is the user's memory allotment updating itself (no accompanying text, just the numbers) and this is a true blue live region.  My suggestion is for the user to set all live regions to off and save the bookmarks/politeness levels.

this sounds reasonable to me.  This  is a really cool feature.  Thanks much for implementing it.

Marked as fixed.