GNOME Bugzilla – Bug 400729
[requirement] Orca should provide support for general audio
Last modified: 2015-07-23 21:36:42 UTC
Orca needs to allow scripts to play sounds. This is a fundamental feature that can be used by other requirements, such as playing audio cues when crossing line boundaries.
Over the weekend, I took a look at pyaudio and its interface to PortAudio: http://people.csail.mit.edu/hubert/pyaudio/ I wasn't thrilled because audio mixing didn't work well for me -- both DECtalk and eSpeak complained and/or the audio wouldn't play at the same time the TTS engine was speaking. It may have been that I didn't get PortAudio and everything set up properly with ALSA, though.
First coarse pass at GNOME 2.24 planning.
Maybe also investigate pyMedia http://pymedia.org/index.html
Alternatively, looks like there are gstreamer bindings for Python: http://pygstdocs.berlios.de/pygst-tutorial/playbin.html gstreamer is pretty popular and the gstreamer python bindings are available on at least Ubuntu and OpenSolaris. On OpenSolaris, they are part of the SUNWgst-python package, which has the following dependency list: SUNWgnome-python-desktop <- SUNWgnome-media-player <- SUNWgst-python SUNWgnome-media-player is also listed as part of the 'entire' package for OpenSolaris, so I think we might be safe in assuming SUNWgst-python is installed on OpenSolaris by default. I played a little bit with this in parallel with eSpeak and DECtalk running and things play/speak simultaneously, meaning the device contention is also not an issue. So, I think the framework might be in place for this one now. This might be a fun thing for someone to take a look at for GNOME 2.28.
Created attachment 135283 [details] [review] Simple patch to test gstreamer with orca This is a quick path to test gstreamer integration with orca.. It plays an ubuntu sound (/usr/share/sounds/question.wav) when orca start. I did this to see if gstreamer works in orca's main loop and such. The code is really bad! This is even not for more then a quick test. My ideas to integrate gstreamer (and sound in general) are as follows: 1. Have a way to generate simple tones of various ferquences like NVDA. This can be used for exampple to inform of progress bar updates (a great NVDA feature in my opion) 2. Play random sounds on events. For example, the pidgin script could play a beep when an user starts writing in a chat. (I could fill different bugs to these suggestions) Regards, Rui Batista
Created attachment 136849 [details] [review] Patch to work with new speech generator formatting strings This patch works off the previous patch and adds a "play" keyword that can be recognized by the speech generator. I had difficulty getting the previous patch to allow me to do anything more than just play an audio file one time (i.e., it would play once and then never ever play again). So, I hacked in an os.system call for now. If you add the following lines to your ~/.orca/user-settings.py file, you should hear a phone ring when you tab between buttons and some other kind of sound when you arrow across icons on the desktop: import orca.formatting import pyatspi orca.settings.sounds = {} orca.settings.sounds[pyatspi.ROLE_PUSH_BUTTON] = "/usr/share/sounds/phone.wav" orca.formatting.formatting['speech']['prefix']['unfocused'] = 'play(role) + ' \ + orca.formatting.formatting['speech']['prefix']['unfocused'] orca.formatting.formatting['speech'][pyatspi.ROLE_ICON]['unfocused'] = \ 'play("/usr/share/sounds/info.wav") + ' \ + orca.formatting.formatting['speech'][pyatspi.ROLE_ICON]['unfocused'] The key element in these lines is "play". The first use (the one that modifies the 'prefix' formatting for speech) says to play an audio file assigned to the role. This is done by looking to the orca.settings.sounds dictionary, which is also set up by the lines above. The second use (the one that overrides the ROLE_ICON formatting string) just specifies a filename directly. Much more could be done with this, such as using audio files to indicate checkbox state, progress bar values, etc.
(In reply to comment #6) > I had difficulty getting the previous > patch to allow me to do anything more than just play an audio file one time > (i.e., it would play once and then never ever play again). So, I hacked in an > os.system call for now. http://pygstdocs.berlios.de/pygst-tutorial/pipeline.html might provide an example of how to do things better using the GStreamer Python API's.
Created attachment 138181 [details] [review] Patch that was committed I committed this patch this morning. It merely adds the ability for people to add audio cues to the formatting strings. That is, it adds unused support for audio so that heavy audio users can help us experiment with how to best integrate audio cues in Orca's presentation. The example in comment #6 still applies. In addition, I was still unable to resolve the issues with using the gstreamer API, so the patch still uses the os.system call.
Created attachment 139194 [details] [review] Patch to set pipe state to null on end of stream so sound plays more then once This patch uses the first alternative in sound.play(), the one that parses the pipeline with gstreamer functions. To play the sound more then once I had to resete the pipe's state to null on end of stream. It is done attaching a callback to the pipe message signal. It is better explained on the pygst tutorial. I leaved there the other alternatives, lets see if this works for someone behond me :) I think the linking problem on the 3rd alternative (one day we should have local branches for our 9997421 ways of having gstreamer not working lol) should be related to dynamic pads. I tried putting an audioconvert bean before the audiosink but that didn't link.
Many thanks for working on the patch! Unfortunately, the patch only seems to allow one sound to be played at a time. For example, make the user-settings.py change provided in comment #6. Then, quickly tab through the Help/OK/Apply/Cancel buttons in the Orca preferences GUI. The telephone ring will only occur intermittently, whereas the telephone ring will occur reliably with the existing code. :-(
I will Here it plays two sounds at the same time... I'm using ubuntu with output set to pulseaudio. Gstreamer supports mixing sounds (through the adder element), maby we should use that not only for playing more sounds at the same but for performance reasons. That way we are not creating audio sinks and opening sound output for each sound that is playing. It seems kind of overkill but with the bunch of sound configurations out there I think we have to have many settings for configuring sound and having just one audio sink will help a bit. But for now whatever that works is fine, even systemcall, we should find the use cases for the thing and implement support for some just to show that works, performance is next lol. Do you think this can be ready for 2.28? I'm asking because saw that this bug is targeted to this release. On other subject I think I can work more seriously on orca general audio support due to an univercity project, will provide more details if they aproove idea :) Regards, Rui Batista
(In reply to comment #11) > configuring sound and having just one audio sink will help a bit. But for now > whatever that works is fine, even systemcall, we should find the use cases for > the thing and implement support for some just to show that works, performance > is next lol. > Do you think this can be ready for 2.28? I'm asking because saw that this bug > is targeted to this release. The system call works well for now and is checked into git master. This went into Orca v2.27.4, so it will be in Orca v2.28.0. > On other subject I think I can work more seriously on orca general audio > support due to an univercity project, will provide more details if they aproove > idea :) This would be awesome! Many thanks!
Related to this bug I filled bug bug #590163 to add sound notifications when a body is typing in pidgin.
We are late in the 2.28 release cycle and I want to focus on "high impact"/"low risk" items that also fall within the release team's restrictions in place. Regretfully, this bug doesn't fit well within those constraints and we'll review it for the 2.29 release cycle.
Created attachment 141762 [details] [review] Another try to get python-gstreamer API working with orca
(In reply to comment #15) > Created an attachment (id=141762) [details] > Another try to get python-gstreamer API working with orca It is working fine in Ubuntu with latest orca and this patch, using the gstreamer API.. plays sounds more then once. Can anyone test it in other os, opensolaris for exampple? Maby the behaviour is different on different systems, want to confirm this :)
Hi, I said on my comments above that I could work on orca general audio support for a univercity work. This is to informe that I'm really able to work on it, it's for an univercity course where we have to make things out of univercity like working with the community, helping people, etc. I talked about orca and they think that is a really good oportunity to develop not just called softskills but also to learn more about opensource in general. It starts on mid-September but I can start working right now. This is more for Will Walker, I have some questions: 1. In first place, may I work on this? 2. Because I add to write a report of what I did and what I've gained with the activity, is there some way to make myself bug assigney or something to proof that I really did it? 3. There are some good ideas and usecases on the above comments and other bugs (for exampple bug 354474). My idea is to organize things, prioritize things, write some documentation... What do you think it is the way to start? should I fill a bug for each usecase and leave this one with just the audio support part? should I create a wiki page to talk about sound support? Sorry for that many questions... Rui Batista
(In reply to comment #16) > (In reply to comment #15) > > Created an attachment (id=141762) [details] [details] > > Another try to get python-gstreamer API working with orca > > It is working fine in Ubuntu with latest orca and this patch, using the > gstreamer API.. plays sounds more then once. > Can anyone test it in other os, opensolaris for exampple? Maby the behaviour is > different on different systems, want to confirm this :) There are several patches in the file: one from 25-Jul and two from 26-Aug. I'm not quite sure which of the three you are proposing -- can you please isolate this down to the one patch you'd like? If you do that, I can test it on OpenSolaris. Thanks for your work!
(In reply to comment #17) > Hi, > > I said on my comments above that I could work on orca general audio support for > a univercity work. > This is to informe that I'm really able to work on it, it's for an univercity > course where we have to make things out of univercity like working with the > community, helping people, etc. > I talked about orca and they think that is a really good oportunity to develop > not just called softskills but also to learn more about opensource in general. > It starts on mid-September but I can start working right now. Yeah! > This is more for Will Walker, I have some questions: > > 1. In first place, may I work on this? You certainly may. Growing our open source community is a goal of mine. > 2. Because I add to write a report of what I did and what I've gained with the > activity, is there some way to make myself bug assigney or something to proof > that I really did it? By creating the patches the way you've done it, you will get credit each time I commit the patches using git-am. In addition, all your comments will appear in the public bug logs, so there should be little doubt who did the work. :-) > 3. There are some good ideas and usecases on the above comments and other bugs > (for exampple bug 354474). My idea is to organize things, prioritize things, > write some documentation... What do you think it is the way to start? should I > fill a bug for each usecase and leave this one with just the audio support > part? should I create a wiki page to talk about sound support? It would be great to start something on the Orca WIKI, say http://live.gnome.org/Orca/Audio, and also open a discussion up on the orca users list. From there, you can develop a set of requirements based upon end user feedback and then design how to meet those requirements in the code. This is going to be exciting!
Created attachment 142178 [details] [review] Isulated patch to try gstreamer working (In reply to comment #18) > (In reply to comment #16) > > (In reply to comment #15) > > > Created an attachment (id=141762) [details] [details] [details] > > > Another try to get python-gstreamer API working with orca > > > > It is working fine in Ubuntu with latest orca and this patch, using the > > gstreamer API.. plays sounds more then once. > > Can anyone test it in other os, opensolaris for exampple? Maby the behaviour is > > different on different systems, want to confirm this :) > > There are several patches in the file: one from 25-Jul and two from 26-Aug. > I'm not quite sure which of the three you are proposing -- can you please > isolate this down to the one patch you'd like? If you do that, I can test it > on OpenSolaris. Thanks for your work! (In reply to comment #18) > (In reply to comment #16) > > (In reply to comment #15) > > > Created an attachment (id=141762) [details] [details] [details] > > > Another try to get python-gstreamer API working with orca > > > > It is working fine in Ubuntu with latest orca and this patch, using the > > gstreamer API.. plays sounds more then once. > > Can anyone test it in other os, opensolaris for exampple? Maby the behaviour is > > different on different systems, want to confirm this :) > > There are several patches in the file: one from 25-Jul and two from 26-Aug. > I'm not quite sure which of the three you are proposing -- can you please > isolate this down to the one patch you'd like? If you do that, I can test it > on OpenSolaris. Thanks for your work! Here it is. Sorry, Git sometimes still confuses me :) I plan to rewirte some things but if it works I'm sure most things will do
(In reply to comment #19) > (In reply to comment #17) > > 1. In first place, may I work on this? > > You certainly may. Growing our open source community is a goal of mine. > Thank you very much, I'm pleased to do this! > > 2. Because I add to write a report of what I did and what I've gained with the > > activity, is there some way to make myself bug assigney or something to proof > > that I really did it? > > By creating the patches the way you've done it, you will get credit each time I > commit the patches using git-am. In addition, all your comments will appear in > the public bug logs, so there should be little doubt who did the work. :-) > Ok, very good. > > 3. There are some good ideas and usecases on the above comments and other bugs > > (for exampple bug 354474). My idea is to organize things, prioritize things, > > write some documentation... What do you think it is the way to start? should I > > fill a bug for each usecase and leave this one with just the audio support > > part? should I create a wiki page to talk about sound support? > > It would be great to start something on the Orca WIKI, say > http://live.gnome.org/Orca/Audio, and also open a discussion up on the orca > users list. From there, you can develop a set of requirements based upon end > user feedback and then design how to meet those requirements in the code. Draft page created, will ask on the orca list latter on and tell my first ideas there so other users can comment. > > This is going to be exciting! I agree!
Comment on attachment 142178 [details] [review] Isulated patch to try gstreamer working I tested this on OpenSolaris and it has the side effect of silencing speech and audio when one lands on a widget enabled for audio. It might be an OpenSolaris/gstreamer integration issue, but the current code that's in git (i.e., the one that does the os.system call) still remains to be the one that works across all platforms. For now, we probably can let the os.system vs. direct gstreamer issue rest and focus on how the use of audio would best be done to enhance the user experience.
(In reply to comment #22) > (From update of attachment 142178 [details] [review]) > I tested this on OpenSolaris and it has the side effect of silencing speech and > audio when one lands on a widget enabled for audio. It might be an > OpenSolaris/gstreamer integration issue, but the current code that's in git > (i.e., the one that does the os.system call) still remains to be the one that > works across all platforms. > Strange :(... I'll install opensolaris on a VM to test on both platforms > For now, we probably can let the os.system vs. direct gstreamer issue rest and > focus on how the use of audio would best be done to enhance the user > experience. Agreed. For simple cases it sufixes altough I guess the performance is not that good... Thanks for testing :)
Hi, The http://live.gnome.org/Orca/Audio has already some ideas of myself and I got from enquires on and off list.. Will, what do you think about these? Some are simple, some are not, some are even very useful but for now I'm kind of brainstorming... Regards, Rui Batista Maby
(In reply to comment #24) > Hi, > > The http://live.gnome.org/Orca/Audio has already some ideas of myself and I got > from enquires on and off list.. > Will, what do you think about these? Some are simple, some are not, some are > even very useful but for now I'm kind of brainstorming... I think this is great! Some of these might be separable from Orca, and I'd strongly encourage trying to push as much stuff as possible upstream from Orca so non-screen reader users could benefit from the work. For example, I could envision a separate utility that tracks the mouse and provides the spatial audio for it. Also, it might be nice to incorporate the "body is typing" audio feature directly into Pidgin. All in all, nice stuff. If possible, I'd also encourage you to engage heavy audio users. T.V. Raman (emacspeak) and Jan Buchal (SpeechDispatcher) come to mind as people that could provide enough suggestions that would take a lifetime to implement. :-)
(In reply to comment #25) > (In reply to comment #24) > > Some of these might be separable from Orca, and I'd strongly encourage trying > to push as much stuff as possible upstream from Orca so non-screen reader users > could benefit from the work. For example, I could envision a separate utility > that tracks the mouse and provides the spatial audio for it. Also, it might be > nice to incorporate the "body is typing" audio feature directly into Pidgin. > Yes, the mouse tracker can be out of orca.. Maby we can get it working in colaboration (like in gnome-mag) but the utility must be a separate program (d-bus or something can be used to communicate with it). I have here a quick prototype (thanks to pyatspi, it just took me one hour), will tun it a bit and release it one of this days so people can give some feedback. Regarding pidgin you're right, a pidgin plugin can do the trick and it will good for many users. > All in all, nice stuff. If possible, I'd also encourage you to engage heavy > audio users. T.V. Raman (emacspeak) and Jan Buchal (SpeechDispatcher) come to > mind as people that could provide enough suggestions that would take a lifetime > to implement. :-) I'll get in touch with them.
(In reply to comment #26) > (In reply to comment #25) > Yes, the mouse tracker can be out of orca.. Maby we can get it working in > colaboration (like in gnome-mag) but the utility must be a separate program > (d-bus or something can be used to communicate with it). I have here a quick > prototype (thanks to pyatspi, it just took me one hour), will tun it a bit and > release it one of this days so people can give some feedback. Just for reference here it is the code (a bit better then yesterday but a pre-alpha-version) http://github.com/ragb/mousetracker Best regards, Rui Batista
Bulk reassigning Will's bugs to the default assignee. (Sorry for the spam!)