GNOME Bugzilla – Bug 652105
[BLOCKED] Implement structural navigation for soffice Writer documents
Last modified: 2015-12-21 21:24:13 UTC
Structural Navigation for HTML documents (Gecko) covers many object types, viz. anchor, block_quote, button, check_box, chunk, combo_box, entry, form_field, heading, landmark, list, list_item, live_region, paragraph, radio_button, separator, table, table_cell, unvisited_link and visited_link. Structural Navigation for Office documents (Soffice) covers only table_cell. Some object types in HTML documents e.g. live region are not relevant in Office documents. Also, AT-SPI does not currently support some element types e.g. list, link etc. in Office documents. However, after exploring AT-SPI support in Office documents I think that structural navigation can extend to cover button, check_box, combo_box, entry, form_field, heading, list box, paragraph, radio_button and table. This requirement also appears in the Orca Roadmap, viz. Improved access to OpenOffice: July 2011 - July 2012 Add the ability to access Writer documents similar to Gecko documents: Structural navigation for moving amongst elements, including form controls. "List of" dialogs, e.g. to get a list of all headings (not yet implemented for Gecko). Document summary (i.e. of the different elements in the document).
Created attachment 189531 [details] [review] Patch to provide Structural Navigation of form elements in Soffice. Please test this patch and let me know if any changes are required. This patch extends Structural Navigation to button, check_box, combo_box, entry, form_field, heading, list box, paragraph, radio_button and table in Soffice documents. Design issues: 1. The basic structural navigation script (/src/orca/structural_navigation.py) provides the framework for navigating among elements. Functions _findPrevious ByMatchRule() and _findNextByMatchRule() are for applications which implement the collection interface of AT-SPI. Functions _findPreviousByPredicate() and _findNextByPredicate() are for applications like Soffice which do not implement the collection interface. For any element type we need a set of four functions. For button we have: _buttonBindings() Key bindings for button _buttonCriteria() Criteria based on collection interface _buttonPredicate() Alternative criteria _buttonPresentation() Speak/ braille button details The basic script implements these functions for almost all element types. To extend Structural Navigation to Soffice, we need to override these functions in the Soffice Structural Navigation script (/src/orca/scripts/apps/soffice/ structural_navigation.py) wherever required. 2. Soffice document object tree includes at any instant only those elements which are currently visible on the screen. So navigating to elements which are not visible involves not only traversing the tree but also scrolling up or down. Functions _findPreviousObject() and _findNextObject() in Soffice Structural Navigation script use the scroll bar object to achieve this. 3. Soffice document elements combo box, heading, list box do not expose the correct AT-SPI roles. So to identify these elements we have to use alternate criteria in functions _comboBoxPredicate(), headingPredicate() and _listBoxPredicate() in Soffice Structural Navigation script.
(In reply to comment #1) > Created an attachment (id=189531) [details] [review] > Patch to provide Structural Navigation of form elements in Soffice. Thanks Dattatray! The thing is: That is a *bunch* of code to hack around some bugs in LibreOffice. Wouldn't it be better to cause those LibreOffice bugs to be fixed? If you (or someone) were to have done that instead: 1. Much of what you have done here would not be necessary. 2. ATs other than Orca would also be able to interact with the document objects more effectively. > 2. Soffice document object tree includes at any instant only those elements > which are currently visible on the screen. So navigating to elements which are > not visible involves not only traversing the tree but also scrolling up or > down. Functions _findPreviousObject() and _findNextObject() in Soffice > Structural Navigation script use the scroll bar object to achieve this. That sounds like another bug in LibreOffice to me. An AT should not have to physically scroll the document just to see what accessible objects are present. In addition, while I haven't tried it yet, I have to wonder if physically manipulating the GUI might have unexpected side-effects for people who use Orca with some other AT because those ATs might be paying attention to AT-SPI events which get emitted as a result of physically scrolling the document. Seems like a bad idea to me. Please open a bug against LibreOffice. > 3. Soffice document elements combo box, heading, list box do not expose the > correct AT-SPI roles. Please see above. Broken ATK implementations are broken. We really need LibreOffice to fix the bugs in question.
>> 2. Soffice document object tree includes at any instant only those elements >> which are currently visible on the screen. So navigating to elements which are >> not visible involves not only traversing the tree but also scrolling up or >> down. Functions _findPreviousObject() and _findNextObject() in Soffice >> Structural Navigation script use the scroll bar object to achieve this. > > That sounds like another bug in LibreOffice to me. An AT should not have to > physically scroll the document just to see what accessible objects are present. > In addition, while I haven't tried it yet, I have to wonder if physically > manipulating the GUI might have unexpected side-effects for people who use Orca > with some other AT because those ATs might be paying attention to AT-SPI events > which get emitted as a result of physically scrolling the document.>> > > Seems like a bad idea to me. Please open a bug against LibreOffice. Thanks. Please do not think that I posted about the patch on orca-list in spite of your feedback. I saw your review after I posted to orca-list. Bugs are already filed against Open Office (bug#117542) and Libre Office (bug#35652).
I mean Open Office bug 117542 and Libre Office bug 35652.
I really mean OOobug 117542 and LibreOfficebug 35652.
Comment on attachment 189531 [details] [review] Patch to provide Structural Navigation of form elements in Soffice. Copying and pasting my reply to Dattatray's latest email on the orca-list advocating for the acceptance of this solution. ~~~~~~~~~~~~ > I wish a few users test the patch before it is discarded. It worked > fine when I tested. We will not manipulate the GUI (i.e. physically adjust the scrollbar) to work around a bug in an app or toolkit. That's not just a hack. That's a hack with potential side effects, including triggering new AT-SPI events which might cause some other AT (a magnifier, or perhaps something else) to do something unwanted in response to the event. The fact that you did not see such an issue while testing Orca without any additional AT's running doesn't mean the problem does not exist and/or that it will not exist. > It is vital that the patch must not break Orca's performance and > stability, but testing is the best way to check if it does. Stability can also be impacted negatively by introducing a bunch of new code which boils down to a hack. The more code introduced, and the bigger the hack, the greater the likelihood that this will occur. It is vital that this does not happen. > Soffice is a very popular application. Improving its accessibility Bingo, improving its accessibility does not mean hacking in Orca. It means fixing LibreOffice. > will benefit a large number of users. Indeed, users of Orca *and* users of other ATs. Your proposed solution fails to do that because it fails to really address the problem at its source. > This particular enhancement is also a part of the Orca roadmap. The roadmap (which I co-authored) says nothing about hacking around this problem. Achieving that roadmap item can -- and may have to -- include our fixing the LibreOffice bugs. Or finding someone who can fix them. > So I wrote some code to work round the Soffice bugs. Not some. Quite a bit. Which, again, manipulates the on-screen GUI objects. > I feel this is in line with Orca methodology of having one default > script to handle the common functions and a special script for each > nonconforming application. The intended purpose is not to handle non-conforming applications. It's to provide different task-specific functionality. What one needs in a chat app is different from what one needs in a spreadsheet app is different from what one needs in a calendaring app, and so on. Yes, we have also used the scripts to deal with non-conforming applications and toolkits. And we have -- and still are -- paying the price. Speaking of which: Last month we had a week-long "ATK/AT-SPI hackfest" at Igalia in Spain. Stakeholders from many different groups attended. The consensus is to do what it takes to bring about a much more unified, standardized implementation of ATK across toolkits and applications rather than placing this burden on ATs like Orca. It will not happen tomorrow, but I am confident it will happen. What you have done is a step away from that much-needed goal; not a step towards it.
Updating the summary to be a summary rather than a specific list, and indicating that this work is blocked.
Created attachment 192242 [details] [review] Add structural navigation for headings and tables to soffice. This commit partially addresses Gnome bug # 652105. Currently Structural Navigation in Soffice covers table cells only. This commit extends it to headings and table. Soffice document object tree includes at any instant only those elements which are currently visible on the screen. However, all paragraphs (visi- ble and invisible) are strung together with the FLOWS_TO and FLOWS_FROM relationships. All headings are paragraphs with paragraph-style:Heading. And each table cell has a paragraph as its child. Therefore, we can traverse all headings and tables using the FLOWS_TO and FLOWS_FROM relationships. These relationships are also utilized by the sayAll() function which reads the whole document.
Dear Jonie Keeping in mind your helpful and precise feedback (comment 6) I have submitted a new patch (comment 8) on 19 July. It is cleaner (doesn't touch GUI objects) and much shorter (a quarter of its predecessor). Please have a look. Regards.
Dattatray: Having had some discussions with the LibreOffice folks, I anticipate that we will have all (including the role-related) issues resolved satisfactorily on their end before too terribly long. That would render your patch unnecessary. I would very, very much prefer to see this done correctly rather than add hacks that I then must turn around and remove. Thank you.
Created attachment 222720 [details] [review] orca: Extend structural navigation to heading, paragraph, table and image An year after submitting the bug to Open Office and Libre Office bug trackers, there is no sign of a solution at the Soffice end. In the meanwhile, my patch (comment 8) was tested at a computer training program for visually disabled (http://www.cdacmumbai.in/index.php/news_and_events/computer_training_program_for_visually_challenged). It worked fine and users found it useful. I understand that the ideal solution would be to fix the bug at Soffice end, not an Orca patch. But the feedback received at the above mentioned event makes me eager to make my solution available to Visually Disabled community, even if temporarily. Is there a way to do this, say by converting my patch into a plugin? If a developer wants to test it, here is the revised version. It is on branch orca-xdesktop and implements structural navigation for heading, paragraph, table and image.
Created attachment 222729 [details] [review] orca: Structural navigation to heading, paragraph, table, image, form elements An year after submitting the bug to Open Office and Libre Office bug trackers, there is no sign of a solution at the Soffice end. In the meanwhile, my patch (comment 8) was tested at a computer training program for visually disabled (http://www.cdacmumbai.in/index.php/news_and_events/computer_training_program_for_visually_challenged). It worked fine and users found it useful. I understand that the ideal solution would be to fix the bug at Soffice end, not an Orca patch. But the feedback received at the above mentioned event makes me eager to make my solution available to Visually Disabled community, even if temporarily. Is there a way to do this, say by converting my patch into a plugin? If a developer wants to test it, here is the revised version. It is on branch orca-xdesktop and implements structural navigation for heading, paragraph, table, image, and form elements viz. push button, radio button, check box, combo box, list box, and entry.