GNOME Bugzilla – Bug 597001
[testing required] Play Beep on progress bar updates
Last modified: 2016-02-17 19:18:35 UTC
Created attachment 144504 [details] [review] Initial implementation of play beep on progress bar updates. When there is an active progress bar it is possible to inform the user about progress change using beeps of various ferquencies. Higher the pitch higher the progress value. This way the user can be informed in real time about the progress of an operation This usecase is described on http://live.gnome.org/Orca/Audio
(In reply to comment #0) > Created an attachment (id=144504) [details] > Initial implementation of play beep on progress bar updates. > Forgot to say that to use this patch you must enable the playBeepOnProgressBarUpdates setting. So on your orca-customizations.py put the following import orca.settings as settings settings.playBeepOnProgressbarUpdates = True On another note, this patch uses the os.system aproach used on orca.sound.Sound class to play sound via Gstreamer. There is the possibility of beeps beeing played one over another because this way it is hard to stop the first one before playing the next. When we can solve the pygst issues this can be corrected Regards and sorry for the spam
This issue was rised again on the orca list: http://mail.gnome.org/archives/orca-list/2010-April/msg00051.html I tried now the patch and it still merge and works with current master. Can anyone test besides myself?
Changing the summary to reflect the need of testing. Please, people interested in this feature (wich I think is really useful) test it, so I can improove it if needed.
Sorry for the spam. Assigning bugs to make it easier to find unassigned bugs. :-)
Review of attachment 144504 [details] [review]: Rui, I just started looking at this (it doesn't work on OpenSolaris, as you indicated was likely the case). However, doing a man of gst-launch, I see the following: === Please note that gst-launch is primarily a debugging tool for developers and users. You should not build applications on top of it. For applications, use the gst_parse_launch() function of the GStreamer API as an easy way to construct pipelines from pipeline descriptions. === So.... should you be using gst_parse_launch instead? Also, a couple of nits: * frequency not ferquency * lines should be under 80 characters long
In playing around some more with this.... I'm able to play beeps in OpenSolaris using pygst. I'm going to play with this some more, but my current leaning is that we check for pygst and if you don't have it, you don't get beeps. Thoughts?
Created attachment 158363 [details] [review] Works for me in OpenSolaris; not in Ubuntu Rui, does this patch work for you in Ubuntu (or whatever distro you are using)? I'm not hearing any sound on my laptop, other than a very faint buzz, but it does get things working in OpenSolaris (which your patch didn't do I'm afraid). Note I've not yet added the setting to the preferences dialog, and I enabled it by default.
Correction, that patch does work for me in Ubuntu. Yay! I was so focused on the sound, that I hadn't noticed Orca wasn't speaking the updates either. D'oh! We have another bug (bug 556346) where certain progress bars -- especially in Ubuntu -- seem to need "tickling" before we get accessible events for them. If I "tickle" the progress bar via flat reviewing to it, Orca starts speaking the progress bar updates as well as playing the tones. I've now tested this on two different Ubuntu installs (one bare-metal install on a laptop, and one virtual box install). So I think this patch works. We need to add a setting to the Orca preferences dialog (and set its default value to False). I'm happy to do both. But I'd still like some testing....
Sorry for the spam. I need to work on some other things at the moment, but some things occurred to me: We should probably separate out the beeping of progress bars from the speaking of them. A user might want to just have beeps and no speech. Also, the way the check is currently done in default.py doesn't properly update the previous value so beeps get played when they shouldn't.
(In reply to comment #7) > Created an attachment (id=158363) [details] [review] > Works for me in OpenSolaris; not in Ubuntu > > Rui, does this patch work for you in Ubuntu (or whatever distro you are using)? > I'm not hearing any sound on my laptop, other than a very faint buzz, but it > does get things working in OpenSolaris (which your patch didn't do I'm afraid). > > Note I've not yet added the setting to the preferences dialog, and I enabled it > by default. Yes it does. Good to hear it working in open Solaris. Last time we tried using pygst calls, (Bug 400729), according to willie Walker, it didn't work but worked in Ubuntu.. so I wrote my patch using gst-launch, even not beeing the best aproach, but worked everywhere. If this works now in Solaris and Ubuntu I'll provide a clean patch to that bug too. I'll try to find how the set the duration with pygst to replace the sleep call :) Rui
> Yes it does. Good to hear it working in open Solaris. Woo hoo! > best aproach, but worked everywhere. If this works now in Solaris and Ubuntu > I'll provide a clean patch to that bug too. That would be awesome thanks! > I'll try to find how the set the duration with pygst to replace the sleep call > :) This would be even awesomer. :-) I kept looking through the docs and I'm just not seeing it.... I will no doubt feel really dumb once we find out what it is. :-) Anyhoo, if you tackle the above, I'll deal with the preferences, separating this option from the speaking, etc. Thanks!
(In reply to comment #11) > > Yes it does. Good to hear it working in open Solaris. > > Woo hoo! > > > best aproach, but worked everywhere. If this works now in Solaris and Ubuntu > > I'll provide a clean patch to that bug too. > > That would be awesome thanks! > > > I'll try to find how the set the duration with pygst to replace the sleep call > > :) > > This would be even awesomer. :-) I kept looking through the docs and I'm just > not seeing it.... I will no doubt feel really dumb once we find out what it is. > :-) > > Anyhoo, if you tackle the above, I'll deal with the preferences, separating > this option from the speaking, etc. Ok, will do.
If there are multiple progress bars giving us events, and the last progress bar was not the most recent one to update, Orca will speak an indication before the percentage (e.g. "Progress bar 2: 40%"). The current beep scheme doesn't address multiple progress bars at all. What should we be doing here?
(In reply to comment #13) > If there are multiple progress bars giving us events, and the last progress bar > was not the most recent one to update, Orca will speak an indication before the > percentage (e.g. "Progress bar 2: 40%"). > > The current beep scheme doesn't address multiple progress bars at all. What > should we be doing here? That is a good question... I guess neither NVDA solves this, it just beeps normally.... Maybe using pann to beep one progress bar from one speaker and another from the other speaker... but if there are three progress bars it doesn't work... Per Apps we can pann it according to screen localization but that also doesn't give any good int to the user... Really really good problem to solve.... Maybe on the orca list someone can give more suggestions... do you want me to write a messaging asking for it?
Created attachment 158398 [details] [review] latest version This does several things: 1. Moves progress bar related settings to the General page. 2. Separates beeps from verbal updates (i.e. you can have none, either, or both) 3. Reworks large chunks of default.handleProgressBarUpdate (i.e. It's a much bigger change than before. Wasn't my intent; just wound up that way.) And it all still works in OpenSolaris. :-) Questions: 1. Does it still work in Ubuntu? 2. Did I break anything? (I've tested quite a bit, but I might have.) 3. Should we be doing braille flash messages for progress bar updates? (perhaps that's another bug; I think the changes I made here will make doing so easier regardless.) 4. Have you found the magical gst duration setting yet? :-) Lemme know what you think about all this and we'll go from there. (Including deciding what to do about multiple progress bars. But one answer might be that the user should opt to also have speech updates if he/she is often in that situation.)
Ugh. Just caught a silly mistake. Don't test that yet. I'll do another patch.
Created attachment 158400 [details] [review] latest version Sorry about that. Need more caffeine I guess. :-) Anyhoo, please test this one. BTW, note that the beep setting is off by default now. You can enable it (and try different combinations of settings progress bar settings) via the general page of the Orca Preferences dialog. Also note that this adds and changes settings enough that backing up your preferences so that Orca doesn't spit up if you remove the patch might be a good idea. (Not convinced it's necessary; but just in case....)
(In reply to comment #17) > Created an attachment (id=158400) [details] [review] > latest version > > Sorry about that. Need more caffeine I guess. :-) > > Anyhoo, please test this one. BTW, note that the beep setting is off by default > now. You can enable it (and try different combinations of settings progress bar > settings) via the general page of the Orca Preferences dialog. Ok... it beeps but... I tested it with many progress bars on evolution (when retrieving messages) and... it blocked orca! Something like nmap: can't allocate more memory! I guess I know why: when beepping you are creating always a new Beep class, so a new gstreamer pipeline... (line 2935 of default.py). Then when stopping the state is put to gst.STATE_READY and resources are not deallocated... then...then... killall orca && killall at-spi-registrid from an ssh session! Was funny indeed. I put that in my first patch so I got what I deserve. I suppose we must have a Beep instance for any known progress bar, is this possible or desirable? Regarding the duration issue: It seems this audiotestsrc thing as no duration setting... just num-buffers and samplesperbuffer, that determine when to send end of stream... the default is -1 so the sound continues playing forever. You can find these running gst-inspect audiotestsrc Another workaround, to prevent the time.sleep call (wich I suppose blocks the orca loop, am I right?) is using gobject timeouts. I tried here and worked, but memory goes away even faster because of the above. Will submit a patch with this aproach so you can check it out (the gobject timeout thing). > > Also note that this adds and changes settings enough that backing up your > preferences so that Orca doesn't spit up if you remove the patch might be a > good idea. (Not convinced it's necessary; but just in case....) It didn't break anything, but I should have made a backup anyway :) Rui Batista
Created attachment 158401 [details] [review] To add on previous patch: change time.sleep to gobject.timeout_add call, and stop the beep on that timeout. call. This must be applied over joannie's latest patch. Is just another idea, I guess duration setting may be the best way, but how?
Yeah, it's clear we have more docs to read. Sounds like we should only have one pipeline. Oops. <smile>
Okay, so I think what we *might* want to do is have a SoundGenerator which has one (and only one) pipeline. In other words, I don't think we want a Beep instance for any known progress bar.
Rui, you're gonna laugh. Or cry. But hopefully laugh. :-) Once I converted things to what I suggested above, namely a SoundGenerator with a single pipeline, I can remove the sleep hack, etc. and beeps seem to play as expected. Maybe that's why you can't set a delay property: You don't need to. :-) Anyhoo, I'm going to hack at this a bit more and hopefully post a new and better version for you to play with.
Created attachment 158408 [details] [review] new approach: sound generator Okay, this gives us the stuff we had in my previous patch (preferences, etc.), in addition: * We don't sleep to achieve duration :-) (Though I still don't see how to set this as a property. And it's killing me.) * A sound generator has been added * Formatting strings have been added (right now, just for focused progress bars) * We have a single orca-wide pipeline (ditto for sink and audiotestsrc) * We clean up after ourselves when shutting down * Some more options to 'beep' added (defaults to sine still, but tick is cool) * A single keypress interrupts sounds in this case (i.e. the sound is a word substitute for a state, role, value, etc.). We might want to refine that later, but for now I think it's helpful. Rui, I wasn't able to reproduce the problem you were having, but hopefully the above changes will make it go away. Also, I left the previous work on sound stuff as-is. But I'd think that some of that can be similarly converted (i.e. out of speech and speech generator), once we work on what the stuff for this bug should look like.
This clearly isn't going to make it into Monday's release. Retargeting.... BTW Rui, I won't be offended if you tell me what I did was in my last attempt is silly -- or just plain wrong. I am the first to admit that I don't know a thing about gst and was just seeing what I could get working.
(In reply to comment #24) > This clearly isn't going to make it into Monday's release. Retargeting.... > > BTW Rui, I won't be offended if you tell me what I did was in my last attempt > is silly -- or just plain wrong. I am the first to admit that I don't know a > thing about gst and was just seeing what I could get working. Hi, Sorry for being late :) I guess, and this is only a guess :), that it still blocks orca's main thread, the one I suppose the at-spi client loop runs... I'm saying this because it just spekas after any beep is played. Ok, could be a possible behaviour but I guess is not the entended one. At least I can listen to speech at the same time of beeps, This is the behaviour they implemented in NVDA and works good in my opinion. Ok, maybe we should reduce the length of the beeps, with 0.5 they become a bit anoing. Anyway, even with one pipeline (something that I really agree with, creating more pipelines makes new connections to the sound system etc) we should provide ways to have sounds playing side by side. On the other hand we might also provide a way to synchronize speech with sound, but this is not the case for this bug, is for the general audio one :). In my oppinio sound for progress bars is kind of a background thing, it's not like using sound to inform of a checkbox state or something. Not only because usually progress bars run fast but, on a download for instance, the user can be doning other things and still listen to some beeping in background to be aware of the progress... Sorry if I'm not being sufficiently clear... Rui Batista
Thanks Rui. And I think I follow you. As much as I enjoyed my little diversion into gst land, there are a lot of things we need to do in Orca (bug fixing, some refactoring, performance analysis and improvement, etc.) which I think I am far more qualified to do than work on anything with pipelines. <grin> You on the other hand seem to have a better idea about gst stuff. So, it would be super cool if you play around with this some more keeping in mind what we've learned so far. If you can get all the soundy things working, and working in both Linux and OpenSolaris, I'll make any necessary gui changes. What do you think?
Comment on attachment 158408 [details] [review] new approach: sound generator Sorry for the spam. Setting to 'needs-work' to get it off of the 'patches without a response' list. Plus this patch does need work. I was just playing around to see if I could get something functional.
(3.0 Planning Spam-o-rama. Sorry!)
Comment on attachment 158401 [details] [review] To add on previous patch: change time.sleep to gobject.timeout_add call, and stop the beep on that timeout. call. Changing status to review to get it off our 'patches without response' list. In addition, sound is likely going to become a plugin in the New World Order.