GNOME Bugzilla – Bug 426010
Implement Verbalized Links
Last modified: 2008-05-07 16:29:06 UTC
Per the Orca functional specification, we need to add speaking/brailling "link" after any links found in OOo Writer documents.
Created attachment 86634 [details] Simple Sample test documents with various links.
Will and I had a quick look at this, this morning. The "normal" way we handle this in Orca is to look to see if the given object implements the Accessible Hypertext interface. See the .../at-spi/idl/Accessibility_Hypertext.idl file for more details. OOo Writer doesn't seem to implement this. From looking at the sample document with at-poke, the only thing that differentiates those links from the other text is that the text is blue and underlined, which isn't really a perfect indication. Email has been sent to the OOo Accessibility team asking how can we determine if there are hypertext links in an OOo Writer document. More information to hopefully follow...
(Just so this doesn't get lost...) Comments from members of the OOo team: "IIRC, Writer implements AccessibleHypertext only for read-only documents (e.g. Help window)." "IMHO the reason was that there was no API for notifying changes when the link (or the paragraph content?) changes. What's the current status with API?" "If you are referring to "hyperlink added" or "hyperlink removed" like events: there is still no sign of it in http://developer.gnome.org/doc/API/2.0/atk/AtkHypertext.html OTOH: will ORCA cache the Hyperlink table ?"
Are we at a point now where an OOo bug should be filed on this missing functionality, and this bug "blocked" against that one?
Further comment from the OOo Accessibility team: "To clarify In theory we are able to deliver them, but seems we do it only for Read-Only docs, because we can't notify if a hyper link itself changes. There is no API for this, so the work around would be to throw away hyper links (the accessible objects) from a paragraph if the paragraph changes."
There has been an OpenOffice issue just opened on this: http://www.openoffice.org/issues/show_bug.cgi?id=77108 Blocking this bug against it.
Removing target milestone from [blocked] bugs. We have little control over them, so we're better off letting priority and severity be our guide for poking the related components.
Unblocking since the blocking bug has been marked as FIXED.
I've just been investigating this one. The underlying OOo bug is indeed fixed, and it you run Orca and you get a "focus:" event when you are positioned on a line in oowriter which contains one or more links, then the word "link" is spoken after each hypertext link. But... It isn't doing the correct thing as we are moving about in the document. And it looks like a bug in the say[Character, Word, Line] implementations in default.py. Here's what's happening. When we are moving around in the document, the we get "object:text-caret-moved" events and the onCaretMoved() method in StarOffice.py is called. There are a couple checks in there currently for OOo specific things, but in this case, we simple fall-through to the bottom and call onCaretMoved() in default.py. This in turn calls _presentTextAtNewCaretPosition() which in turn will call sayCharacter, sayWord or sayLine. There is nothing in those three methods that checks for hypertext links and speaks "link" after the character/word/line. Will, should I: 1/ Adjust this bug to reflect the new problem? 2/ Close this one out, and open a new bug to fix this in default.py? 3/ Add extra code to the StarOffice.py script to fix this problem there? Maybe overwriting sayCharacter/Word/Line there... 4/ None of the above? Personally I'd like to see this fixed in default.py, because it cuts down the amount of application specific code and doing it there might make it useful to other applications that have a similar problem.
> Personally I'd like to see this fixed in default.py, because it cuts > down the amount of application specific code and doing it there might > make it useful to other applications that have a similar problem. That would be my vote too. It looks like at least Evo has implemented the accessible Hypertext interface. And I would hope that whoever winds up working on Evince's accessible text support would also implement accessible Hypertext at the same time.... So that would bring us to three (counting all of OOo as a single creature). If we could file bugs that would convince Pidgin and Tomboy Notes to do the right thing (i.e. implement accessible Hypertext), all the better. Although.... That might be a gtk bug as gtk-demo's Hypertext doesn't seem to implement the Hypertext interface. :-( Anyhoo, point is there's potentially a lot of hypertext out there for which we want to implement verbalized links.
Just thought of something else: We might want to add something to whereAmI so that doing a whereAmI on a line with a link causes the presence of that link to be indicated. If so, perhaps what you wind up implementing in default.py could be used (or easily transfer) there as well?
There's code in StarOffice.py that I think does what we need. See endOfLink and sayWriterWord. My quick gut impression is that we should be able to just add something like: if self.endOfLink(obj, entity, startOffset, endOffset): speech.speak(_("link")) to the end of sayCharacter(), sayWord() and sayLine() (where 'entity' is the character, word or line in question), and we will have what we want.
I should add that we'll have to put a little more code in endOfLink that's current in the start of locusOfFocusChanged. Something on the order of: try: hypertext = event.source.queryHypertext() except NotImplementedError: return False if hypertext.getNLinks() == 0: return False ... rest of code ... near the start.
(In reply to comment #9) ... > It isn't doing the correct thing as we are moving about in the > document. And it looks like a bug in the say[Character, Word, Line] > implementations in default.py. ... > in default.py. This in turn calls _presentTextAtNewCaretPosition() > which in turn will call sayCharacter, sayWord or sayLine. > > There is nothing in those three methods that checks for hypertext > links and speaks "link" after the character/word/line. ... > Personally I'd like to see this fixed in default.py, because it cuts > down the amount of application specific code and doing it there might > make it useful to other applications that have a similar problem. I agree it should be fixed in default.py. In addition, we're also seeing traffic on the Orca list where links might not be handled in other applications as well, such as Tomboy. So, better addressing the presentation of links in both speech and braille is something we should do. That probably should be done in a separate bug, as long as you agree. I'm fine with repurposing this one if you want to.
First coarse pass at GNOME 2.24 planning.
The spec for verbalized links says: "When moving between lines in an office document and the line contains one or more links orca should speak the word link after the link. If the "link" voice is specified, the link should be read using the link voice. This functionality should also be available in "say all" mode. In braille the link should always be underlined using dots 7 and 8. If the user moves by character or word, the word "link" will not be repeatedly spoken. If the user double presses insert+f to get detailed information on a character they will be informed that the character is part of a link." So the approach mentioned above in comments #12 and #13 on how to fix this is incorrect. This will need to be revisited.
Plan for 2.24 is to just do Insert-f (x2) and simple where-am-I includes "link". Later we'd do say all and flat review (but still as part of this bug).
Created attachment 108932 [details] [review] Prototype patch showing proof-of-concept for braille support for links. Joanie sent me this last night (thank you). I'm attaching it here so it doesn't get lost as I work on other things. This patch adds in braille support for links. There is a new setting in settings.py called brailleLinkIndicator which is currently hardwired to BRAILLE_SEL_BOTH. Joanie chatted with Mike last night about this. Mike, you stated that you'd like settings to be configurable and toggleable. For configurable, are you looking for a new frame/panel on the Braille tab in the Orca preferences window? Something like: Link Indicator o None o Dot 7 o Dot 8 o Dots 7 and 8 with a default setting of "Dots 7 and 8"? For toggleable, are you looking for a key binding for this functionality? If so, what should it be (or is it initially unbound)? Joanie, you also mention that Mike wants to add a checkbox to the list of attributes. I didn't understand what that meant. Can you explain in more detail? Further questions for you Mike: Should there be a similar link configuration on the Speech tab in the Orca preferences dialog? If so, what should it look like? Does this need to be toggleable to? If yes, what keybinding (or is it unbound)? Thanks.
> Prototype patch showing proof-of-concept for braille support for links. Emphasis on "prototype" and "proof-of-concept." ;-) Also note that this doesn't address it for Gecko.py (FF and Thunderbird), but does seem to get the job done for OOo Writer and Evo. For the Gecko stuff, we'll presumably just need to add a check to see if the user wants links to be "underlined" and, if so, add that to the braille generator we already have for links. > For configurable, are you looking for a new frame/panel on the Braille > tab in the Orca preferences window? Something like: > > Link Indicator > o None > o Dot 7 > o Dot 8 > o Dots 7 and 8 > > with a default setting of "Dots 7 and 8"? While I think that would make our lives easier, here's what I took from my conversation with Mike: 1. Links should be treated as if it was just another text attribute (such as bold, italics, etc.) 2. The link indicator should be the same as the indicator for other attributes. In other words, if I used the Preferences dialog to set my attribute indicator to Dot 8, bold, italics, ..., and links would all be "underlined" with Dot 8. We therefore wouldn't need a separate frame/panel in the Preferences dialog. 3. Because we're treating the property of being a link as if it were just another text attribute, we should make it possible for the user to enable it/disable it just like it were any other attribute. (i.e., it would be listed among all the true text attributes and have a checkbox for speech and another for braille on the text attributes page). Here's where I think things *might* get somewhat ugly on the development side of things, but I haven't actually looked at the code w.r.t. this bit yet. I suspect it's doable if we're clever. > For toggleable, are you looking for a key binding for this functionality? > If so, what should it be (or is it initially unbound)? This is the checkbox-in-the-dialog bit rather than a key binding I believe. > Joanie, you also mention that Mike wants to add a checkbox to the list > of attributes. I didn't understand what that meant. Can you explain in more > detail? I hope I've done that in the above. :-) If not, please let me know. Mike, if I misunderstood anything, please comment/correct. Thanks!
BTW, because the nature of "underlining" links and speaking (verbalizing?) them seem to be two separate problems, I opened a new bug specific to the braille half (bug 527022). If we're going to do the work for braille as part of this bug (i.e. bug 426010), I can close out the other as a dup.
In looking at this problem in a similar light to other things we may need to handle (e.g., verbalized capitals, speaking 'misspelled' automatically, different voices for different things, etc.), I think we might have a bigger refactor to consider after 2.24. So, here's a proposal for how to handle links in any application for 2.24: 1) With speech, the word "link" will be spoken after we speak the text for a link. This will not be spoken when moving character-by-character or word-by-word through a link, but is more for when we speak a single line of text and/or when a link gets focus (i.e., you tabbed to it). When moving from line to line and a link crosses line boundaries, we will treat the broken text as though each portion were an individual link (i.e., we would speak "link" on one line and "link" on another). 2) Insert+f will always speak the word "link" if the thing is a link. Where Am I will also include this. Mike - we need to know where in the order of things the word "link" should be spoken. 2) With braille, the "Braille" page of the preferences UI will have a "Hyperlink Indicator" subpanel identical to the "Selection Indicator" subpanel. These dots will be used on the braille display to indicate a character is associated with a link. 3) If "None" is selected for the "Hyperlink Indicator", the word "link" will be brailled after the link text. If anything except "None" is chosen, the associated dots will be used for each character associated with the link. I talked about this loosely with Mike, and I think he's good with this. The things I didn't ask him about were the name of the subpanel and what to do when crossing line boundaries. So, if there's disagreement about that, we should clear it up. In addition, I think we also need to clarify what is needed for speech when we left/right arrow *to* a link in the middle of text.
Created attachment 108954 [details] [review] Patch to add in the ORCA_KEY-f functionality. > 2) Insert+f will always speak the word "link" if the thing is a link. > Where Am I will also include this. Mike - we need to know where in > the order of things the word "link" should be spoken Just want to make sure I'm understanding what is required here. This patch will add the speaking of "link" after any text attributes have been spoken, when the user presses ORCA_KEY-f. As far as I can see, there is no double-press ORCA_KEY-f, so I just did this for a single-press ORCA_KEY-f. If we want the speaking of "link" to be anywhere other than either before or after the text attribute list, then the patch will get much more involved. For the where-am-I change, doing a KP_Enter in oowriter, returns a paragraph. Are we looking to add the word "link" in (in the link voice), after every hypertext link for the whole paragraph here? Or just for the word under where the text caret is currently set? If it's the former, I'm confused by needing to know where in the order of things "link" should be spoken. Clarification would be greatly appreciated on this too. Thanks.
(In reply to comment #22) > Created an attachment (id=108954) [edit] > Patch to add in the ORCA_KEY-f functionality. > > > 2) Insert+f will always speak the word "link" if the thing is a link. > > Where Am I will also include this. Mike - we need to know where in > > the order of things the word "link" should be spoken > > Just want to make sure I'm understanding what is required here. > > This patch will add the speaking of "link" after any text > attributes have been spoken, when the user presses ORCA_KEY-f. > > As far as I can see, there is no double-press ORCA_KEY-f, so I > just did this for a single-press ORCA_KEY-f. This seems fine to me. If you're thinking what I'm thinking, the double-press is for the detailed Where Am I summary. That would be a whole separate RFE for OOo, I think. > If we want the speaking of "link" to be anywhere other than > either before or after the text attribute list, then the > patch will get much more involved. At the end sounds fair to me: "bold link" vs. "link bold". > For the where-am-I change, doing a KP_Enter in oowriter, > returns a paragraph. Are we looking to add the word "link" > in (in the link voice), after every hypertext link for the > whole paragraph here? Or just for the word under where the > text caret is currently set? If it's the former, I'm confused > by needing to know where in the order of things "link" should > be spoken. If there are no links in the paragraph, what do we currently present?
> If there are no links in the paragraph, what do we currently present? I created an oowriter document with a single line (paragraph) containing: The quick brown fox jumps over the lazy dog. I underlined it all. I then ser "quick" to Bold as well. Single-press KP_ENTER: BRAILLE LINE: 'soffice Application Untitled1 - OpenOffice.org Writer Frame Untitled1 - OpenOffice.org Writer RootPane ScrollPane Document view The quick brown fox jumps over the lazy dog. $l' VISIBLE: 'The quick brown fox jumps over t', cursor=1 SPEECH OUTPUT: '' SPEECH OUTPUT: 'paragraph' SPEECH OUTPUT: 'The quick brown fox jumps over the lazy dog.' text utterances=[''] SPEECH OUTPUT: '' ---- Double-press KP_ENTER: BRAILLE LINE: 'soffice Application Untitled1 - OpenOffice.org Writer Frame Untitled1 - OpenOffice.org Writer RootPane ScrollPane Document view The quick brown fox jumps over the lazy dog. $l' VISIBLE: 'The quick brown fox jumps over t', cursor=1 SPEECH OUTPUT: '' SPEECH OUTPUT: 'paragraph' SPEECH OUTPUT: ' ; underline The ; bold underline quick ; underline brown fox jumps over the lazy dog.' text utterances=[''] SPEECH OUTPUT: '' So it looks like it already processes the line word by word. It should be possible to adjust that to put "link" in after words that are hyper text links as well. Is that what we want? Also, is it the word in the line that gets spoken in the link voice, or is it the "link" word after, or both?
The double press for where am I is a whole different thing I'm currently specing. I like placing "link" at the end of the text sort of like selected.
> The double press for where am I is a whole different thing I'm currently > specing. I like placing "link" at the end of the text sort of like selected. Mike, in oowriter, where-am-I will "process" a whole paragraph. Not all the paragraph may be a link. Placing "link" at the end might not be the best solution.
(In reply to comment #24) > > If there are no links in the paragraph, what do we currently present? .... > So it looks like it already processes the line word by word. It should be > possible to adjust that to put "link" in after words that are hyper text > links as well. Is that what we want? For Where Am I, I'd guess we'd want to treat the text that is output just as we'd treat it if you arrowed to the line. So, this would mean putting the word "link" after the phrase that represents the link. Conceptually, I think what we're kind of doing is treating the link as an object embedded in the text and we're appending its rolename to the presentation of the link, and we need to handle this for regular navigation, say all, and flat review. Eeks, this one might end up being pretty hard. > Also, is it the word in the line that gets spoken in the link voice, or > is it the "link" word after, or both? Ah - you raise a good point. I think that if it is present and different from the default voice, the hyperlink voice would ideally be used whenever presenting anything to do with links. So, this would mean using the hyperlink voice when navigating across characters and/or words of a link, the "link" word it self, up/down arrowing across lines, etc. When reading a line that is composed of both link and non-link text, I think it would be desirable to speak the link text with the hyperlink voice and the non-link text with the default voice (unless it is all upper case, of course). The same would apply to say all and flat review. I think, however, we may be pretty incomplete in the 'use the hyperlink voice' department as well as in the 'use the uppercase voice' department. So, those might be separate bugs to open, but I'm not quite sure of the exact description for them and I think we might have a bit of refactoring to do to handle them.
It just dawned on me that at least for Firefox, whereAmI does special stuff for links such as attempt to ascertain (and present) the type of link. Is this something we also want to (attempt to) do in other applications?
If such information is exposed to us in other applications it would be really nice to alert the user to the different kinds of links. I think we should use firefox as the baseline for all link information.
Created attachment 109648 [details] [review] Revision A. Patch hopefully implements the following based on the specification by Will in comment #21: * Insert+f will always speak the word "link" if the thing is a link. * Where-am-I also includes speaking "link" for hypertext links. * The "Braille" page of the preferences UI has a "Hyperlink Indicator" frame identical to the "Selection Indicator" subpanel. These dots will be used on the braille display to indicate a character is associated with a link. * If anything except "None" is chosen, the associated dots will be used for each character associated with the link. Some notes and questions: * The where-am-I presentation of "link" is based on a double press of the KP_Enter key (same as for text attributyes). Is that what we want? I've currently added it just after where the text attributes are spoken. Is that what we want? * Per a phone conversation with Will just before I went on vacation, we are currently (for GNOME 2.24) not going to use the "hypertext voice" when presenting these links. * I used the same keyboard accelerators on the Braille page of the preferences UI for the hyperlink indicator radio buttons as are used for the Selection indicator. This will mean that the user has to enter them twice to get to the hyperlink radio buttons. If this isn't desired, what alternatives should be used? * I noticed that I'd occasionally get an "index out of range" traceback in the new linkIndicator code in getAttributeMask in braille.py. I've added a try/except block around it to catch these. Don't know why they are happening at the moment. Still to be implemented: * For braille hyperlink indicator, if "None" is selected for the "Hyperlink Indicator", the word "link" will be brailled after the link text. * With speech, the word "link" will be spoken after we speak the text for a link. This will not be spoken when moving character-by-character or word-by-word through a link, but is more for when we speak a single line of text and/or when a link gets focus (i.e., you tabbed to it). When moving from line to line and a link crosses line boundaries, we will treat the broken text as though each portion were an individual link (i.e., we would speak "link" on one line and "link" on another). Before I take this too much further (and to save me having to potentially implement this multiple times), it would help if Mike could answer the questions that are outstanding from Will, plus the new ones I raise above. With that information, I can hopefully add in the remainder of the changes in just a couple more revisions. Thanks.
Created attachment 109709 [details] Revised OOo writer document with extra link examples.
Created attachment 109714 [details] [review] Revision B. Implements: * For braille hyperlink indicator, if "None" is selected for the "Hyperlink Indicator", the word "link" will be brailled after the link text. One possible concern is when the hypertext link is just part of a word. There are a couple of cases in the new version of the sample OOo writer document that have that. In these cases, there won't be a space between "link" and the following non-hyperlink text. If we try to handle this, it starts to get murky because the following non-hypertext characters might be just punctuation and you don't really want to add in a space there. Personally I think we should just ignore these edge cases but I'll leave it to Mike to decide what's needed here.
Here's a summary of some of the things we decided at the Orca team meeting this afternoon w.r.t. the outstanding questions/concerns. * When we are crossing line boundaries for a large link, we will say "link" for each line. * We will initially have two identical sets of keyboard accelerators for the Braille Selection indicator radio buttons and the Braille Link indicator radio buttons. * Mike will ask the community whether we should present "link" on the braille display after hyper-text links. Other possibilities are to present nothing, if the user has set the Braille Link indicator to None, or to add another radio button to that group for "Add Link to Braille Line" (needs better wording). Revision B of the patch currently has us presenting "link" on the braille display if the user has selected None. This will be adjusted once we hear back from the Orca user community. If we are still to present "link" on the braille display after hypertext links, then we should add an extra bit of code to check if the character after the link is white space or punctuation. If it's not, then we should add in a space. * Presenting the link information as part of the double-press where-am-I information is fine with Mike. * The position of the "link" in the where-am-I information is fine with Mike. * When moving line by line the speaking of "link" will be similar to the way that the where-am-I double-press function does it. In other words, we will speak "link" just before each hypertext link. This is still to be implemented (I'll start looking at it tomorrow). Let me know if I've missing anything.
> * When moving line by line the speaking of "link" will be similar to > the way that the where-am-I double-press function does it. In other > words, we will speak "link" just before each hypertext link. This is > still to be implemented (I'll start looking at it tomorrow). This troubles me as it is opposite of what we currently do in firefox. Currently in firefox we speak the word link after the link text when arrowing by line. This is by design and the way it should stay.
Mike, just so I understand (and before I try to implement it tomorrow): For the default case (which will therefore affect Evolution and OOo), do you want the word "link" spoken before the actual text or after it? If it's after it, it'll be different then the way that the where-am-I double-press function currently speaks it.
I'd like it spoken after to keep with the firefox functionality. I'm OK with it being different than WhereAmI as that is expected.
Created attachment 109780 [details] [review] Revision C. New patch implements the following: 1/ Adds "link" after any hyper text links for what is being spoken by the sayLine() method in default.py. I created a new adjustForlinks() method in default.py to do this and the braille code calls this as well. 2/ If the link was not followed by a whitespace or punctuation character, then add in a space to make it more presentable. I was get a Traceback of "index out of range" when processing hypertext links for blank (empty) lines in OOo. This was with an object:text-caret-moved event that had a detail1 of -1, such as: OBJECT EVENT: object:text-caret-moved detail=(-1,0) app.name='soffice' name='None' role='paragraph' state='editable enabled focusable multi line multiselectable showing visible' relations='flows to' The line was '' (length 0), but doing: hyperText = obj.queryHypertext() nLinks = hyperText.getNLinks() resulted in nLinks = 1. Should we be processing them? Or maybe it's YAOOB. I've added a check into the adjustForLinks() routine to make sure that link.endIndex < len(line), so the traceback is no longer happening, but it would be nice to know if we can ignore these events. Pylints to 10.0 (after fixing other peoples boo boo's; changes included in this patch). I think that's all I can do on this bug until Mike has determined what the user community want w.r.t. brailling "link" after any hyper text links.
Created attachment 110124 [details] [review] Revision D. Revised patch (no brailling "link" for None setting) committed to SVN trunk. Moving to "[pending]".