After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 400729 - [requirement] Orca should provide support for general audio
[requirement] Orca should provide support for general audio
Status: RESOLVED OBSOLETE
Product: orca
Classification: Applications
Component: general
2.17.x
Other All
: Low enhancement
: FUTURE
Assigned To: Orca Maintainers
Orca Maintainers
post-3.0
Depends on:
Blocks: 354474
 
 
Reported: 2007-01-25 18:55 UTC by Willie Walker
Modified: 2015-07-23 21:36 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Simple patch to test gstreamer with orca (1.06 KB, patch)
2009-05-24 22:06 UTC, Rui Batista
none Details | Review
Patch to work with new speech generator formatting strings (6.09 KB, patch)
2009-06-17 16:27 UTC, Willie Walker
none Details | Review
Patch that was committed (7.59 KB, patch)
2009-07-10 11:28 UTC, Willie Walker
committed Details | Review
Patch to set pipe state to null on end of stream so sound plays more then once (2.10 KB, patch)
2009-07-25 01:44 UTC, Rui Batista
needs-work Details | Review
Another try to get python-gstreamer API working with orca (6.84 KB, patch)
2009-08-26 15:38 UTC, Rui Batista
none Details | Review
Isulated patch to try gstreamer working (3.40 KB, patch)
2009-09-01 00:19 UTC, Rui Batista
rejected Details | Review

Description Willie Walker 2007-01-25 18:55:02 UTC
Orca needs to allow scripts to play sounds.  This is a fundamental feature that can be used by other requirements, such as playing audio cues when crossing line boundaries.
Comment 1 Willie Walker 2007-05-22 14:19:53 UTC
Over the weekend, I took a look at pyaudio and its interface to PortAudio:

http://people.csail.mit.edu/hubert/pyaudio/

I wasn't thrilled because audio mixing didn't work well for me -- both DECtalk and eSpeak complained and/or the audio wouldn't play at the same time the TTS engine was speaking.  It may have been that I didn't get PortAudio and everything set up properly with ALSA, though.
Comment 2 Willie Walker 2008-03-11 14:06:47 UTC
First coarse pass at GNOME 2.24 planning.
Comment 3 Rich Burridge 2008-06-03 17:15:40 UTC
Maybe also investigate pyMedia
http://pymedia.org/index.html
Comment 4 Willie Walker 2009-02-17 17:10:34 UTC
Alternatively, looks like there are gstreamer bindings for Python:

http://pygstdocs.berlios.de/pygst-tutorial/playbin.html

gstreamer is pretty popular and the gstreamer python bindings are available on at least Ubuntu and OpenSolaris.  On OpenSolaris, they are part of the SUNWgst-python package, which has the following dependency list:

SUNWgnome-python-desktop <- SUNWgnome-media-player <- SUNWgst-python

SUNWgnome-media-player is also listed as part of the 'entire' package for OpenSolaris, so I think we might be safe in assuming SUNWgst-python is installed on OpenSolaris by default.

I played a little bit with this in parallel with eSpeak and DECtalk running and things play/speak simultaneously, meaning the device contention is also not an issue.

So, I think the framework might be in place for this one now.  This might be a fun thing for someone to take a look at for GNOME 2.28.

Comment 5 Rui Batista 2009-05-24 22:06:30 UTC
Created attachment 135283 [details] [review]
Simple patch to test gstreamer with orca

This is a quick path to test gstreamer integration with orca.. It plays an ubuntu sound (/usr/share/sounds/question.wav) when orca start. I did this to see if gstreamer works in orca's main loop and such. The code is really bad! This is even not for more then a quick test.

My ideas to integrate gstreamer (and sound in general) are as follows:
1. Have a way to generate simple tones of various ferquences like NVDA. This can be used for exampple to inform of progress bar updates (a great NVDA feature in my opion)
2. Play random sounds on events. For example, the pidgin script could play a beep when an user starts writing in a chat.
(I could fill different bugs to these suggestions)

Regards,

Rui Batista
Comment 6 Willie Walker 2009-06-17 16:27:39 UTC
Created attachment 136849 [details] [review]
Patch to work with new speech generator formatting strings

This patch works off the previous patch and adds a "play" keyword that can be recognized by the speech generator.  I had difficulty getting the previous patch to allow me to do anything more than just play an audio file one time (i.e., it would play once and then never ever play again).  So, I hacked in an os.system call for now.

If you add the following lines to your ~/.orca/user-settings.py file, you should hear a phone ring when you tab between buttons and some other kind of sound when you arrow across icons on the desktop:

import orca.formatting
import pyatspi
orca.settings.sounds = {}
orca.settings.sounds[pyatspi.ROLE_PUSH_BUTTON] = "/usr/share/sounds/phone.wav"
orca.formatting.formatting['speech']['prefix']['unfocused'] = 'play(role) + ' \
    + orca.formatting.formatting['speech']['prefix']['unfocused']
orca.formatting.formatting['speech'][pyatspi.ROLE_ICON]['unfocused'] = \
    'play("/usr/share/sounds/info.wav") + ' \
    + orca.formatting.formatting['speech'][pyatspi.ROLE_ICON]['unfocused']

The key element in these lines is "play".  The first use (the one that modifies the 'prefix' formatting for speech) says to play an audio file assigned to the role.  This is done by looking to the orca.settings.sounds dictionary, which is also set up by the lines above.

The second use (the one that overrides the ROLE_ICON formatting string) just specifies a filename directly.

Much more could be done with this, such as using audio files to indicate checkbox state, progress bar values, etc.
Comment 7 Willie Walker 2009-07-10 00:24:33 UTC
(In reply to comment #6)
> I had difficulty getting the previous
> patch to allow me to do anything more than just play an audio file one time
> (i.e., it would play once and then never ever play again).  So, I hacked in an
> os.system call for now.

http://pygstdocs.berlios.de/pygst-tutorial/pipeline.html might provide an example of how to do things better using the GStreamer Python API's.
Comment 8 Willie Walker 2009-07-10 11:28:38 UTC
Created attachment 138181 [details] [review]
Patch that was committed

I committed this patch this morning.  It merely adds the ability for people to add audio cues to the formatting strings.  That is, it adds unused support for audio so that heavy audio users can help us experiment with how to best integrate audio cues in Orca's presentation.

The example in comment #6 still applies.  In addition, I was still unable to resolve the issues with using the gstreamer API, so the patch still uses the os.system call.
Comment 9 Rui Batista 2009-07-25 01:44:37 UTC
Created attachment 139194 [details] [review]
Patch to set pipe state to null on end of stream so sound plays more then once

This patch uses the first alternative in sound.play(), the one that parses the pipeline with gstreamer functions. To play the sound more then once I had to resete the pipe's state to null on end of stream. It is done attaching a callback to the pipe message signal. It is better explained on the pygst tutorial.

I leaved there the other alternatives, lets see if this works for someone behond me :)

I think the linking problem on the 3rd alternative (one day we should have local branches for our 9997421 ways of having gstreamer not working lol) should be related to dynamic pads. I tried putting an audioconvert bean before the audiosink but that didn't link.
Comment 10 Willie Walker 2009-07-27 16:05:36 UTC
Many thanks for working on the patch! 

Unfortunately, the patch only seems to allow one sound to be played at a time.  For example, make the user-settings.py change provided in comment #6.  Then, quickly tab through the Help/OK/Apply/Cancel buttons in the Orca preferences GUI.  The telephone ring will only occur intermittently, whereas the telephone ring will occur reliably with the existing code. :-(
Comment 11 Rui Batista 2009-07-27 16:49:48 UTC
I will

Here it plays two sounds at the same time... I'm using ubuntu with output set to pulseaudio. 

Gstreamer supports mixing sounds (through the adder element), maby we should use that not only for playing more sounds at the same but for performance reasons. That way we are not creating audio sinks and opening sound output for each sound that is playing. It seems kind of overkill but with the bunch of sound configurations out there I think we have to have many settings for configuring sound and having just one audio sink will help a bit. But for now whatever that works is fine, even systemcall, we should find the use cases for the thing and implement support for some just to show that works, performance is next lol.
Do you think this can be ready for 2.28? I'm asking because saw that this bug is targeted to this release.

On other subject I think I can work more seriously on orca general audio support due to an univercity project, will provide more details if they aproove idea :)

Regards,

Rui Batista


Comment 12 Willie Walker 2009-07-27 17:40:25 UTC
(In reply to comment #11)
> configuring sound and having just one audio sink will help a bit. But for now
> whatever that works is fine, even systemcall, we should find the use cases for
> the thing and implement support for some just to show that works, performance
> is next lol.
> Do you think this can be ready for 2.28? I'm asking because saw that this bug
> is targeted to this release.

The system call works well for now and is checked into git master.  This went into Orca v2.27.4, so it will be in Orca v2.28.0.

> On other subject I think I can work more seriously on orca general audio
> support due to an univercity project, will provide more details if they aproove
> idea :)

This would be awesome!  Many thanks!
Comment 13 Rui Batista 2009-07-29 19:15:08 UTC
Related to this bug I filled bug bug #590163 to add sound notifications when a body is typing in pidgin.
Comment 14 Willie Walker 2009-08-14 15:44:00 UTC
We are late in the 2.28 release cycle and I want to focus on "high impact"/"low risk" items that also fall within the release team's restrictions in place.  Regretfully, this bug doesn't fit well within those constraints and we'll review it for the 2.29 release cycle.
Comment 15 Rui Batista 2009-08-26 15:38:46 UTC
Created attachment 141762 [details] [review]
Another try to get python-gstreamer API working with orca
Comment 16 Rui Batista 2009-08-26 15:42:48 UTC
(In reply to comment #15)
> Created an attachment (id=141762) [details]
> Another try to get python-gstreamer API working with orca

It is working fine in Ubuntu with latest orca and this patch, using the gstreamer API.. plays sounds more then once.
Can anyone test it in other os, opensolaris for exampple? Maby the behaviour is different on different systems, want to confirm this :)
Comment 17 Rui Batista 2009-08-31 19:16:21 UTC
Hi,

I said on my comments above that I could work on orca general audio support for a univercity work.
This is to informe that I'm really able to work on it, it's for an univercity course where we have to make things out of univercity like working with the community, helping people, etc.
I talked about orca and they think that is a really good oportunity to develop not just called softskills but also to learn more about opensource in general.
It starts on mid-September but I can start working right now.

This is more for Will Walker, I have some questions:

1. In first place, may I work on this?

2. Because I add to write a report of what I did and what I've gained with the activity, is there some way to make myself bug assigney or something to proof that I really did it?

3. There are some good ideas and usecases on the above comments and other bugs (for exampple bug 354474). My idea is to organize things, prioritize things, write some documentation... What do you think it is the way to start? should I fill a bug for each usecase and leave this one with just the audio support part? should I create a wiki page to talk about sound support?

Sorry for that many questions...

Rui Batista
Comment 18 Willie Walker 2009-08-31 20:58:49 UTC
(In reply to comment #16)
> (In reply to comment #15)
> > Created an attachment (id=141762) [details] [details]
> > Another try to get python-gstreamer API working with orca
> 
> It is working fine in Ubuntu with latest orca and this patch, using the
> gstreamer API.. plays sounds more then once.
> Can anyone test it in other os, opensolaris for exampple? Maby the behaviour is
> different on different systems, want to confirm this :)

There are several patches in the file: one from 25-Jul and two from 26-Aug.  I'm not quite sure which of the three you are proposing -- can you please isolate this down to the one patch you'd like?  If you do that, I can test it on OpenSolaris.  Thanks for your work!
Comment 19 Willie Walker 2009-08-31 21:02:45 UTC
(In reply to comment #17)
> Hi,
> 
> I said on my comments above that I could work on orca general audio support for
> a univercity work.
> This is to informe that I'm really able to work on it, it's for an univercity
> course where we have to make things out of univercity like working with the
> community, helping people, etc.
> I talked about orca and they think that is a really good oportunity to develop
> not just called softskills but also to learn more about opensource in general.
> It starts on mid-September but I can start working right now.

Yeah!

> This is more for Will Walker, I have some questions:
> 
> 1. In first place, may I work on this?

You certainly may.  Growing our open source community is a goal of mine.

> 2. Because I add to write a report of what I did and what I've gained with the
> activity, is there some way to make myself bug assigney or something to proof
> that I really did it?

By creating the patches the way you've done it, you will get credit each time I commit the patches using git-am.  In addition, all your comments will appear in the public bug logs, so there should be little doubt who did the work.  :-)

> 3. There are some good ideas and usecases on the above comments and other bugs
> (for exampple bug 354474). My idea is to organize things, prioritize things,
> write some documentation... What do you think it is the way to start? should I
> fill a bug for each usecase and leave this one with just the audio support
> part? should I create a wiki page to talk about sound support?

It would be great to start something on the Orca WIKI, say http://live.gnome.org/Orca/Audio, and also open a discussion up on the orca users list.  From there, you can develop a set of requirements based upon end user feedback and then design how to meet those requirements in the code.

This is going to be exciting!
Comment 20 Rui Batista 2009-09-01 00:19:25 UTC
Created attachment 142178 [details] [review]
Isulated patch to try gstreamer working

(In reply to comment #18)
> (In reply to comment #16)
> > (In reply to comment #15)
> > > Created an attachment (id=141762) [details] [details] [details]
> > > Another try to get python-gstreamer API working with orca
> > 
> > It is working fine in Ubuntu with latest orca and this patch, using the
> > gstreamer API.. plays sounds more then once.
> > Can anyone test it in other os, opensolaris for exampple? Maby the behaviour is
> > different on different systems, want to confirm this :)
> 
> There are several patches in the file: one from 25-Jul and two from 26-Aug. 
> I'm not quite sure which of the three you are proposing -- can you please
> isolate this down to the one patch you'd like?  If you do that, I can test it
> on OpenSolaris.  Thanks for your work!

(In reply to comment #18)
> (In reply to comment #16)
> > (In reply to comment #15)
> > > Created an attachment (id=141762) [details] [details] [details]
> > > Another try to get python-gstreamer API working with orca
> > 
> > It is working fine in Ubuntu with latest orca and this patch, using the
> > gstreamer API.. plays sounds more then once.
> > Can anyone test it in other os, opensolaris for exampple? Maby the behaviour is
> > different on different systems, want to confirm this :)
> 
> There are several patches in the file: one from 25-Jul and two from 26-Aug. 
> I'm not quite sure which of the three you are proposing -- can you please
> isolate this down to the one patch you'd like?  If you do that, I can test it
> on OpenSolaris.  Thanks for your work!

Here it is. Sorry, Git sometimes still confuses me :)

I plan to rewirte some things but if it works I'm sure most things will do
Comment 21 Rui Batista 2009-09-01 00:42:06 UTC
(In reply to comment #19)
> (In reply to comment #17)
> > 1. In first place, may I work on this?
> 
> You certainly may.  Growing our open source community is a goal of mine.
> 

Thank you very much, I'm pleased to do this!

> > 2. Because I add to write a report of what I did and what I've gained with the
> > activity, is there some way to make myself bug assigney or something to proof
> > that I really did it?
> 
> By creating the patches the way you've done it, you will get credit each time I
> commit the patches using git-am.  In addition, all your comments will appear in
> the public bug logs, so there should be little doubt who did the work.  :-)
> 

Ok, very good.

> > 3. There are some good ideas and usecases on the above comments and other bugs
> > (for exampple bug 354474). My idea is to organize things, prioritize things,
> > write some documentation... What do you think it is the way to start? should I
> > fill a bug for each usecase and leave this one with just the audio support
> > part? should I create a wiki page to talk about sound support?
> 
> It would be great to start something on the Orca WIKI, say
> http://live.gnome.org/Orca/Audio, and also open a discussion up on the orca
> users list.  From there, you can develop a set of requirements based upon end
> user feedback and then design how to meet those requirements in the code.

Draft page created, will ask on the orca list latter on and tell my first ideas there so other users can comment.

> 
> This is going to be exciting!

I agree!
Comment 22 Willie Walker 2009-09-01 16:31:23 UTC
Comment on attachment 142178 [details] [review]
Isulated patch to try gstreamer working

I tested this on OpenSolaris and it has the side effect of silencing speech and audio when one lands on a widget enabled for audio. It might be an OpenSolaris/gstreamer integration issue, but the current code that's in git (i.e., the one that does the os.system call) still remains to be the one that works across all platforms.

For now, we probably can let the os.system vs. direct gstreamer issue rest and focus on how the use of audio would best be done to enhance the user experience.
Comment 23 Rui Batista 2009-09-01 18:42:32 UTC
(In reply to comment #22)
> (From update of attachment 142178 [details] [review])
> I tested this on OpenSolaris and it has the side effect of silencing speech and
> audio when one lands on a widget enabled for audio. It might be an
> OpenSolaris/gstreamer integration issue, but the current code that's in git
> (i.e., the one that does the os.system call) still remains to be the one that
> works across all platforms.
> 
Strange :(... I'll install opensolaris on a VM to test on both platforms

> For now, we probably can let the os.system vs. direct gstreamer issue rest and
> focus on how the use of audio would best be done to enhance the user
> experience.
Agreed. For simple cases it sufixes altough I guess the performance is not that good...

Thanks for testing :)
Comment 24 Rui Batista 2009-09-09 02:36:18 UTC
Hi,

The http://live.gnome.org/Orca/Audio has already some ideas of myself and I got from enquires on and off list..
Will, what do you think about these? Some are simple, some are not, some are even very useful but for now I'm kind of brainstorming... 

Regards,

Rui Batista
Maby
Comment 25 Willie Walker 2009-09-10 20:03:47 UTC
(In reply to comment #24)
> Hi,
> 
> The http://live.gnome.org/Orca/Audio has already some ideas of myself and I got
> from enquires on and off list..
> Will, what do you think about these? Some are simple, some are not, some are
> even very useful but for now I'm kind of brainstorming... 

I think this is great!

Some of these might be separable from Orca, and I'd strongly encourage trying to push as much stuff as possible upstream from Orca so non-screen reader users could benefit from the work.  For example, I could envision a separate utility that tracks the mouse and provides the spatial audio for it.  Also, it might be nice to incorporate the "body is typing" audio feature directly into Pidgin.

All in all, nice stuff.  If possible, I'd also encourage you to engage heavy audio users.  T.V. Raman (emacspeak) and Jan Buchal (SpeechDispatcher) come to mind as people that could provide enough suggestions that would take a lifetime to implement.  :-)
Comment 26 Rui Batista 2009-09-13 21:15:01 UTC
(In reply to comment #25)
> (In reply to comment #24)
> 
> Some of these might be separable from Orca, and I'd strongly encourage trying
> to push as much stuff as possible upstream from Orca so non-screen reader users
> could benefit from the work.  For example, I could envision a separate utility
> that tracks the mouse and provides the spatial audio for it.  Also, it might be
> nice to incorporate the "body is typing" audio feature directly into Pidgin.
> 

Yes, the mouse tracker can be out of orca.. Maby we can get it working in colaboration (like in gnome-mag) but the utility must be a separate program (d-bus or something can be used to communicate with it). I have here a quick prototype (thanks to pyatspi, it just took me one hour), will tun it a bit and release it one of this days so people can give some feedback.

Regarding pidgin you're right, a pidgin plugin can do the trick and it will good for many users.


> All in all, nice stuff.  If possible, I'd also encourage you to engage heavy
> audio users.  T.V. Raman (emacspeak) and Jan Buchal (SpeechDispatcher) come to
> mind as people that could provide enough suggestions that would take a lifetime
> to implement.  :-)

I'll get in touch with them.
Comment 27 Rui Batista 2009-09-15 00:58:06 UTC
(In reply to comment #26)
> (In reply to comment #25)
> Yes, the mouse tracker can be out of orca.. Maby we can get it working in
> colaboration (like in gnome-mag) but the utility must be a separate program
> (d-bus or something can be used to communicate with it). I have here a quick
> prototype (thanks to pyatspi, it just took me one hour), will tun it a bit and
> release it one of this days so people can give some feedback.

Just for reference here it is the code (a bit better then yesterday but a pre-alpha-version)
http://github.com/ragb/mousetracker

Best regards,
Rui Batista
Comment 28 Joanmarie Diggs (IRC: joanie) 2010-04-03 20:23:57 UTC
Bulk reassigning Will's bugs to the default assignee. (Sorry for the spam!)