One of the hangs was attempting to keep the application list up to date in Orca by automatically building it whenever various at-spi events were received (e.g., children added and removed from the desktop).  This has been removed and a getKnownApplications method has been added to at-spi for scripts that want to get the list of applications.

Orca can still hang, however, when making a call to with the at-spi and gnome-speech interfaces. I'm not sure of the exact conditions for this, or if it's even predictable.  There's some work being done in the at-spi infrastructure to provide multiple threads for handling at-spi events from applications, and this might help some.

I put some experimental threading.Timer code into orca.py and focus_tracking_presenter.py to try to detect hangs, but I wasn't satified with the results.  It's still in there (commented out) for further experimentation, though.

Created attachment 66100 [details] [review]
Patch to initialize the ORB in threaded mode

This is based upon what we've learned in bug 342614.  It initializes the ORB in threaded mode.  With this patch in place, I slammed on gedit while being forced to watch that sappy show, American Idol.  I couldn't get it to hang, but that's only a data point.  Intermittent problems are a bear to find.  But, this patch is at least based upon some analysis.

Created attachment 67485 [details] [review]
Patch to move script processing of input events to the gidle handler

The processing of input events, such as keyboard and braille commands to control Orca itself, can often result in AT-SPI calls buy scripts.  This patch moves the processing of input events by scripts to the gidle handler, which is where we tend to do all of our AT-SPI calls.

One the remaining things to do is to refactor speech so things like SayAll and speech progress handling are also done in the gidle handler.  With the refactor of speech, the bulk of Orca logic will be done in a single thread, helping prevent deadlocks as much as possible.

I modified the Orca signal handlers to dump a stack trace.  The way this can be used is as follows:

1) Enable orca.debug.debugLevel=orca.debug.LEVEL_ALL
2) Run Orca, wait for a hang.
3) From a virtual console or other means, find the orca process and
   kill -TERM it.  You might need to do this several times - what you're
   looking for is a stack trace to be put at the end of debug.out.

The stack trace above can give some insight.  For example, I discovered that the gnome-speech driver for DECtalk was hanging on a call to TextToSpeechReset, which caused Orca's call to speech.stop to hang, which then hung Orca.
 

Created attachment 69667 [details] [review]
Patch to abort Orca if a hang is detected.

This patch puts a timer in place when performing calls that might take a long time.  If the timer expires, orca will abort.

We've done a lot of work here and have eliminated many hangs.  We've also added defensive code to Orca itself to quit if it detects a hang.  The last remaining badness seems to be in two areas:

1) DECtalk hanging.  This is not an Orca bug and the defensive code we added will at least detect this, allowing Orca to be restarted.

2) Flat review hangs.  These are covered in other bugs.

If any more hangs occur, we will enter specific bugs for them.