Tracking.

I submitted a trouble ticket (http://trac.galago-project.org/ticket/91) for the notification-daemon with the Galago Project.

Ticket #91 (defect)

Opened 2 seconds ago
notification-daemon panel not accessible to people with disabilities
Status: new
Reported by: 	lmonsanto       Assigned to: 	chipx86
Priority: 	high 	        Milestone: 	
Component: 	notification-daemon 	Version: 	
Severity: 	normal 	        Keywords: 	
Cc: 		                D-BUS Version: 	
Patch Included: 	0 	OS/Distro Version: 	

The message panel used by the notification-daemon is not accessible on Ubuntu. Using the at-poke utility, the panel has no accessible (e.g., text) subcomponents. It appears that the theme engine simply renders the text onto the panel. This bug results in applets, like the battery monitor, not being accessible to people with disabilities. When a notification is displayed to a person with a visual disability, a screen reader, like Orca or Gnopernicus, should speak the notification message (e.g., low battery power") to the user. Similarly, a Braille device would present the message to a blind user. One solution is to use an accessible GTK+ widget to display the message.

This defect has high-priority since people with visual disablities are not aware when a notification is displayed. In the case of low battery power, the laptop to shut down unexpectedly before a disabled user to take remedial action.

Add accessibility keyword.  Apologies for spam.

Here's the current bug status from http://trac.galago-project.org/ticket/91:

    * severity changed from normal to major.
    * milestone set to notification-daemon 1.0.

We are now getting AT-SPI events. I haven't examined the events to see whether they have sufficient information for Orca, but I know we can (at least) say that the Battery Charge Monitor has popped up. I now need to verify I can get the warning test in the popup.

Here's my entry in the Galago bug tracking tool:

=======================================================================

02/13/07 09:11:37: Modified by lmonsanto

I downloaded and built the notification-daemon from http://galago-project.org/downloads.php. Are there any other galago components that I should build and install?

[~] ls -l /usr/libexec/notification-daemon -rwxr-xr-x 1 root root 102935 2007-02-06 10:46 /usr/libexec/notification-daemon*

[~] ps -elf | grep notification-daemon 0 S monsanto 10862 1 0 75 0 - 11641 1 Feb12 ? 00:00:00 /usr/libexec/notification-daemon

I test by unplugging the power cable from the back of my laptop which runs Ubuntu 6.10 (Edgy). I immediately get a popup in the lower-right-hand corner of the screen telling me that the laptop is now on battery power. The at-poke tool shows that I am now getting AT-SPI events when the Battery Charge Monitor pops up.

generic event 'object:bounds-changed' generic event 'object:bounds-changed' A|co|ac:embedded component:Battery Charge Monitor:Monitor a lap...(0) (0)

This is definitely good! I now need to verify that the event information is sufficient for Orca.

Thanks for all your work! Lynn

Lynn, it would be really useful to know sooner than later if we are now getting useful information for these messages.  I got bit by this one again the other day when my battery went dead and shut down my laptop with no warning.  If we are getting something good I'll spec this for you asap.  

Created attachment 82639 [details] [review]
Adds support for gnome-power-manager

This is a partial fix, in that it adds support for the gnome-power-manager. It is not sufficient, in that other scripts will probably need to be created for other important apps that use the galago notification-daemon.

This looks very promising.  Is the app.toolkitName something unique to Galago?  If so, it might be possible to write a toolkit script to handle the applications that use it (like we did for Gecko.py).

If we develop a general solution, we should make it configurable. For example, I just started Thunderbird 3 and it displayed a list of new messages using (I assume) the notification daemon. You can turn this off in Thunderbird 3 preferences, but this could be pretty annoying to a new user, It's kind of like pop-up ad windows in Firefox. These can also be disabled in Firefox, but there was a time when you couldn't.

BTW, I will dig deeper and see if there is something that identifies the notification-daemon when we get an event.

Hi Lynn.  I wanted to give this a try but I seem to be unable to install libgalago.  I realize you're not galago tech support :-)  but I thought you might have an idea....  I am doing:

./autogen.sh --prefix=/usr (no problem)
make (no problem)
sudo make install (gets pretty far....)

Making install in po
make[1]: Entering directory `/home/jd/galago/libgalago/po'
/bin/sh `case "@MKINSTALLDIRS@" in /*) echo "@MKINSTALLDIRS@" ;; *) echo "../@MKINSTALLDIRS@" ;; esac` /usr/share
/bin/sh: Can't open ../@MKINSTALLDIRS@
make[1]: *** [install-data-yes] Error 2
make[1]: Leaving directory `/home/jd/galago/libgalago/po'
make: *** [install-recursive] Error 1

Thanks in advance!!!

Lynn:  Orca has several things it does when it attempts to find a script:

1) It first looks for an application-specific script
2) It then looks for a toolkit-specific script
3) It finally falls back to default.py

For #2, our sole example is Gecko.py.  Gecko is the name of the toolkit used for Thunderbird and Firefox, and it is exposed to Orca as "Gecko."  Thus, we can write a Gecko.py script to handle the majority of what Orca needs to handle both Firefox and Thunderbird.

The question I'm wondering about is if we see something similar for applications that are Galago based.  When you look at the app.toolkitName of the source of an event from one of these apps that presents a balloon type message, does it have something that uniquely identifies it as a Galago app?  If so, we could write a toolkit-specific script that would cover Galago-based applications.  This script would be separate from Thunderbird, Firefox, etc., and would only be used for applications that are Galago based.  Again, this is only if the app.toolkitName gives us something that helps us to uniquely identify this as a Galago-based app.

(In reply to comment #11)
> Hi Lynn.  I wanted to give this a try but I seem to be unable to install
> libgalago.  I realize you're not galago tech support :-)  but I thought you
> might have an idea....  I am doing:
> 
> ./autogen.sh --prefix=/usr (no problem)
> make (no problem)
> sudo make install (gets pretty far....)
> 
> Making install in po
> make[1]: Entering directory `/home/jd/galago/libgalago/po'
> /bin/sh `case "@MKINSTALLDIRS@" in /*) echo "@MKINSTALLDIRS@" ;; *) echo
> "../@MKINSTALLDIRS@" ;; esac` /usr/share
> /bin/sh: Can't open ../@MKINSTALLDIRS@
> make[1]: *** [install-data-yes] Error 2
> make[1]: Leaving directory `/home/jd/galago/libgalago/po'
> make: *** [install-recursive] Error 1
> 
> Thanks in advance!!!
> 

I don't believe you need to reinstall galago.  Just unplug your power cord, if you are running a laptop, and orca should speed a message saying that you are now on battery power.   Please see my next comment below.

Hi Lynn.  Thanks!

At the time I wrote the comment, unplugging the power cord did nothing.  Perhaps some of the latest Feisy updates solved that?  (Who knows....)  Anyhoo, Orca definitely says something now, but not quite that I'm running on battery power; just that it's "recalculating information..." From debug.out:

vvvvv PROCESS OBJECT EVENT object:bounds-changed vvvvv
OBJECT EVENT: object:bounds-changed                    detail=(0,0)
---------> QUEUEING EVENT object:children-changed:add
    app.name='gnome-power-manager' name=None role='panel' state='ENABLED RESIZABLE SENSITIVE SHOWING VISIBLE' relations=''
Looking for script at orca-scripts.gnome-power-manager.py...
...could not find orca-scripts.gnome-power-manager.py
Looking for script at scripts.gnome-power-manager.py...
...found scripts.gnome-power-manager.py
gnome-power-manager.py: __init__
NEW SCRIPT: gnome-power-manager (module=orca.scripts.gnome-power-manager)
gnome-power-manager.py: onBoundsChanged
gnome-power-manager.py: 'object:bounds-changed' src='panel'
gnome-power-manager.py: _searchChildrenForDisplayedText
gnome-power-manager.py: _getObjText: text='', desc='Recalculating information...', label='None'
gnome-power-manager.py:  role='panel', text='Recalculating information...'
SPEECH OUTPUT: 'Recalculating information...'
gnome-power-manager.py: _searchChildrenForDisplayedText
gnome-power-manager.py: _getObjText: text='', desc='', label='None'
gnome-power-manager.py:    role='icon', text=''
^^^^^ PROCESS OBJECT EVENT object:bounds-changed ^^^^^

As Mike reported, Orca speaks the first time the power cord is removed but is silent afterwards as you battery power runs down.   The problem is that Orca receives a bound-changed event every time the Gnome Power Manager pops up a message.  The obj.description contains the message text.  The problem is that the description continues to contain the same  text, even as the displayed messages change telling you that you have less and less power.  The obj.description is cached, since one would not expect it to change during the lifetime of an object.  obj.accessible.description also remains the same.  I tried to find a python binding for getAccessibleContext, but couldn't.   Hopefully, there is  a method to find the new description.

The Battery Status Monitor displays tool tips with the appropriate information, but Orca cannot process tool tip events without the battstat process crashing.   It might be possible to query battstat directly for the battery state, but I haven't tried yet. 

I need to spend the next couple days getting baseline code coverage data for Orca, and I'll return to the problem as soon as possible.

Any ideas?

(In reply to comment #14)
> Hi Lynn.  Thanks!
> 
> At the time I wrote the comment, unplugging the power cord did nothing. 
> Perhaps some of the latest Feisy updates solved that?  (Who knows....)  Anyhoo,
> Orca definitely says something now, but not quite that I'm running on battery
> power; just that it's "recalculating information..." From debug.out:

Hi Joanie,

Thanks for the problem report.  Battery state reporting is not right, and needs to be fixed.  It's really an important feature.

Lynn

Created attachment 83146 [details] [review]
Additional fix

Prevents the caching of object accessible descriptions when the gnome-power-manager is running. This forces the power manager to provide a new accessible descriptionn containing the latest power management information. This is really a temporary solution, since it turns off description caching for all objects and won't scale a general solution. A per-script caching policy would be useful.

I submitted a request for the galago notification-daemon to send a state-changed:visible event, instead of all the bounds-changed events, when the gnome-power-manager notification is displayed.

http://trac.galago-project.org/ticket/91

There is another issue with the gnome-power-manager not displaying notifications when battery power is just about gone. I need to investigate the problem more on Feisty before submitting a bug report on the problem.

Thanks Lynn!

> There is another issue with the gnome-power-manager not displaying
> notifications when battery power is just about gone. I need to investigate the
> problem more on Feisty before submitting a bug report on the problem.

I've been playing with this on my machine for the past couple days.  I wonder if it's a gnome-power-manager problem or if it's just a battery-gone-bad problem.  The reason being is that I made a point of watching my battery status every few minutes.  For the past few times where my machine has just powered down, I had checked just about one minute prior and the battery monitor said my battery had well over an hour left on it.  Then, suddenly and without warning, my machine did a graceful power down.

I recall seeing this same behavior when my PowerBook's battery went south.  One second, the battery status monitor told me I had nothing to worry about.  The next second, the machine says "See you later, I'm checking out."

*** Bug 410088 has been marked as a duplicate of this bug. ***

A couple things:

1. On Feisty, Orca is now speaking "recalculating" instead of the message that is displayed in the balloon help.

2. I haven't received a reply to my last request to have the notification daemon fire a 'state change' event where I can get the balloon help contents. I need to figure out how to send email to chipx86.

Removing target milestone from [blocked] bugs.  We have little control over them, so we're better off letting priority and severity be our guide for poking the related components.

Lynn:  We *may* now be getting appropriate events out of the notification daemon (e.g. window-activate, state-changed:visible, etc.).  Just stumbled across them in a debug.out on my Gutsy laptop.

Thanks for the debug.out. It looks a 'notification-daemon.py' script might be able to get the message text from the 'onStateChanged' event.source.description, or the AccessibleDescription of a window child. I'll try this when Gutsy is more stable and can be run on my laptop.

Here is the second message I sent to chipx86 about the problem. I did not receive a reply to my first email.

-------- Original Message --------
Subject: 	accessibility problem with notification-daemon
Date: 	Tue, 19 Jun 2007 09:37:42 -0700
From: 	Lynn MonSanto <Lynn.MonSanto@sun.com>
To: 	chipx86@chipx86.com
CC: 	Willie Walker <William.Walker@Sun.COM>, Lynn Monsanto <monsanto@sun.com>


Hi Chip,

I hope I'm sending this email to the right person, and you're the person 
who responded to my galago ticket #91

  http://trac.galago-project.org/ticket/91

Can you take a look at my last comment requesting that the 
notification-daemon fire an event when balloon help is visible and once 
again when the balloon help becomes non-visible.

Here is the bug report for the Orca screen reader concerning the problem:

 http://bugzilla.gnome.org/show_bug.cgi?id=354479

Thanks!
Lynn Monsanto

Created attachment 90504 [details] [review]
The start of a notification daemon script.

This is a script that reads out the notification "bubble" as it arrives. There are three things that need further work:

1. Usability - Right now it just interrupts Orca with new speech. Is this the desired effect?
2. Input focus - The notification window could not receive keyboard input, at least not easily. So in the meantime this solution only works with notifications that time out and disappear, and do not need an action performed by the user. We could program around this constraint by using the action interfaces on the button accessibles, but again, someone needs to think about how this could be usable.
2.5. Since the window is not easily focusable, it is sort of traceless through the screen reader, meaning if the screen reader starts reading it's contents via this script, and is inturupted by pressing a key or whatever, there is no way of returning to the notification and exploring it.
3. At least on my system (feisty) curiously enough the notification daemon is not started with the GAIL module, making it inaccessible. I was able to work around this by killing the daemon and restarting it from the command line. I suspect that the session dbus daemon was not started with proper environment variables.

Eitan:  This looks like the gaim script.  Is this above the correct attachment?

Created attachment 90563 [details] [review]
Notification daemon script

Oops, don't know how that other attachment happened.

1) What should happen with (In reply to comment #26)
> Created an attachment (id=90504) [edit]
> The start of a notification daemon script.

Looks like a good addition.  There are a couple methods in default.py that a script will inherit that might be useful (though maybe not): findByRole and findUnrelatedLabels. findByRole finds all objects under a given root that are of a given role, and findUnrelatedLabels finds all labels under a given root that are not bound to any object.

When adding strings for translation (text = _('Notification %s') % ' '.join(texts)), please make sure to add documentation for them so translators know what to do.  You should hopefully be able to find examples of the pattern we use to notify translators throughout the code.  In addition, I wonder if "Notification" is actually needed.  That is, would just the text suffice? Mike?

> This is a script that reads out the notification "bubble" as it arrives. There
> are three things that need further work:

I'll let Mike address the usability questions, as well as advise what we should do for braille.

> 3. At least on my system (feisty) curiously enough the notification daemon is
> not started with the GAIL module, making it inaccessible. I was able to work
> around this by killing the daemon and restarting it from the command line. I
> suspect that the session dbus daemon was not started with proper environment
> variables.

Very strange.  How is the notification daemon started (e.g., who starts it)?

Thanks!

Created attachment 90625 [details] [review]
Notification daemon script

Thanks Willie for that input, the changes here use your suggestions.

The "notification-daemon" process is spawned by the session d-bus daemon when it is first required, meaning when a client application first uses libnotify.

I have a gut feeling that the notification daemon used o be accessible on my desktop, maybe in Edgy, or maybe I did some bad changes here. Could anyone confirm this? Just run Accerciser, you should see the notification-daemon in the applications list.

> I have a gut feeling that the notification daemon used o be accessible on my
> desktop, maybe in Edgy, or maybe I did some bad changes here. Could anyone
> confirm this? Just run Accerciser, you should see the notification-daemon in
> the applications list.

It's not in my applications list in Feisty or Gutsy initially.  In Feisty, it never is.  In Gutsy doing something to cause it to be launched (e.g. switching to battery power) will add it to  the applications list.  So perhaps it's just broken in Feisty??

Yes, the behavior you are getting in Gutsy is correct.
Something is broken in Feisty.

Thanks.

I don't yet have a functional gutsy so it'll be a bit of time before I can test this one.  I'm downloading a gutsy CD now so we'll see what happens.

Mike,

You don't need Gutsy to test this.

In a terminal run:
killall notification-daemon
/usr/lib/notification-daemon/notification-daemon &

This should make notifications accessible.

Created attachment 90684 [details]
Notification test script

This is a test script that pops up a notification.

There are a few other things that could be tweaked and tested, like urgency, notification type, etc.

This should do for now.

Eitan,

We discussed this bug in todays's Orca staff meeting, and decided your proposed patch should be committed. A couple other things need to be done to remove the current solution:

1. remove src/orca/scripts/gnome-power-manager.py.

2. remove the following two lines from "orca/src/orca/default.py" starting at line line 702:

        listeners["object:bounds-changed"]                  = \
            self.noOp

Lynn

Eitan,

As requested by Will, I've reassigned this bug to you, so all the changes are done by one person.

Thanks!
Lynn

Created attachment 90711 [details] [review]
Notification daemon script

Here is an updated patch with Lynn's input.

Like I said on my first patch comments, there is a usability question on how to deal with notifications with actions. Or how to explicitly close notifications that never time out. There are button widgets with accessible action interfaces, but there is no real option for focus.

> 
> Like I said on my first patch comments, there is a usability question on how to
> deal with notifications with actions. Or how to explicitly close notifications
> that never time out. There are button widgets with accessible action
> interfaces, but there is no real option for focus.
Hopefully when Li Yuan finishes his work on adding keyboard access to the notification area this problem will be addressed. 
> 

I found the relevant Ubuntu bug that makes notification daemon inaccessible, and added a comment:
https://bugs.launchpad.net/control-center/+bug/62163

Also, a better workaround for Edgy users is deleting the line "use-session-dbus" in /etc/X11/Xsession.options.

This does seem to work well.  I'm taking my name off this unless we find problems in gnome 2.20. 

In http://trac.galago-project.org/ticket/91, chipx86 indicates this might be a gnome-power-manager problem.

All said and done, though, if the notification daemon script Eitan wrote provides the equivalent access that the user needs, maybe we can just close this bug.  Thoughts?  Mike?

I think this is fine.  We still have the problem of not being able to navigate to these controls but that is being tracked by another bug.

Executive summary: The notification area and notification daemon are two separate things, they are unrelated, although they do integrate nicely. We have no solution for the notification area, we have a very partial solution for the notification daemon.

1. The notification area in the panel is inaccessible, it can never gain focus.
The notification area usually provides tiny icons that hint at the status of something, for example with the power manager puts a tiny battery there when we are running on battery power, the batteries current charge level could be estimated by how green or red the battery picture is. There is also a a tooltip that shows up when you put the pointer over it that shows a percentage of charge and estimated time left on battery. Of course since the notification area in not keyboard navigable the tooltip is useless if you don't use a pointer device.

As far as I know anything in the notification area is completely inaccessible today, just to get an idea, I have right now in the notification area:
 a. A bluetooth status icon.
 b. A Pidgin status icon.
 c. A network manager applet (how could you even access network manager outside this area?).
 d. Tomboy notes. It shows no status, but I could click on it for quick access to notes.
 e. Power manager status, a picture of a battery or a plug, depending on whether my laptop is plugged in.

2. The notification daemon "balloons" are inaccessible, they could never gain focus.
The notification daemon is a seperate component from the notification area. Applications could send messages to this daemon, and they will be displayed either as "balloons" that seem to pop out of there respective notification area icon,to give the user context. Or as toaster popups: when a new message comes in  it pops up at the bottom right of the screen, if an older message is still there, it pushes it up, and so you get a stack of recent messages. This is the main reason why this is implemented as a daemon, so messages from discrete applications could play nice with each other. Usually these messages have a tiny "x" you could press to dismiss them, they will also typically time-out and disappear. A client application could define various degrees of urgency to the message. By convention, a very urgent message will actually never time out and will stay visible until it is dismissed. 

Some notifications have buttons on them that define related actions. For example if I update my kernel in Ubuntu, a restart will be required, so a message balloon comes in that tells me this and gives be two buttons: "reboot now" and "dismiss".

The idea of these notifications is that they are unobtrusive: they never grab focus. Of course this is what makes them useless to Orca too. The current Orca script does one very simple thing: it reads a message as it pops up. The user has no ability of dismissing the message or of performing any of it's related actions because it cannot receive focus via keyboard. Furthermore, while a sighted user could read the message as many times as she wishes, a blind user will only hear it once when Orca speaks it via this script. After Orca announces it once, it's lost forever.

Phew!

Thanks for the summary Eitan!  It seems like we might need to file another bug to provide better keyboard traversal of the notification area.  With this, users could better access the history and such and also interact with buttons that might appear in the area.  Does this seem right to you?

(In reply to comment #46)
> Thanks for the summary Eitan!  It seems like we might need to file another bug
> to provide better keyboard traversal of the notification area.  With this,
> users could better access the history and such and also interact with buttons
> that might appear in the area.  Does this seem right to you?
> 

Sorry - already filed.  See bug 431030 - [blocked] Cannot use keyboard to navigate gnome-panel notification area.  Given that that issue is being tracked, and since the notification-daemon script seems to solve the specific subject of this bug, I think we probably can close this bug as FIXED.  Thoughts?

Closing as FIXED (see comment #47).  Thanks!

We have a regression here. The script is never activated.

Since we started selectively listening to events (essentially, since the pyatspi migration).

Created attachment 125529 [details] [review]
Proposed fix

This adds a listener to default.py for the window:create event with a noOp callback.

Thanks Eitan!  Committed to trunk and gnome-2-24.