GNOME Bugzilla – Bug 105962
Magnifier should track caret position using text-caret-moved events not text-changed events
Last modified: 2004-12-22 21:47:04 UTC
It was discovered that when using the magnifier with Java Applications on GNOME (Eg. StylePad from JFC demos) the magnifier wasn't tracking the fact that the text caret had moved within the text area to a place that wasn't visible in the magifier. Originally, this was though to be a bug in the JABG, which it was, as regards text-changed events not having the "insert/delete/replace" detail information (for which there is a fix in progress). On further investigation it was decided that this shouldn't have been a problem since text-caret-moved events are still being generated correctly and that the magnifier should be tracking these not the text-changed events.
I tried to reproduce this bug, but I don't know if I use the right application. With SwingSet Java application, using JEditPane I cannot reproduce it. But, with mozilla this bug appears because in case of mozilla text object, function AccessibleText_getCharacterExtents returns wrong values for position and dimension of specified character. Please tell me where can I file a bug for mozilla, accessibility part. Also tell me more exactly how can I reproduce it (what is the name of the application and how can I get it).
Remus: The bug is with the way gnopernicus listens to events from text widgets, you should not need to use a particular app to fix this bug. gnopernicus should be listening to and responding to 'text caret' events rather than text-change events when it does magnifier/caret tracking. It appears that gnopernicus is currently using text-changed events for caret tracking, which is not correct; "object:text-caret-moved" events are the events which you should be listening for when tracking the caret with the magnifier.
Currently, gnopernicus is listening AND text-caret-moved too and generates caret-moved event (this can be checked using a gtk application where this feature works ok. So, in my opinion not this is the problem.
thanks Remus, we will look further and see if we can determine more about the situations where this problem occurs.
For mozilla bug 193818 is registered for this situation. (bugzilla.mozilla.org)
Remus: note that this bug was logged concerning general gnopernicus behavior and behavior with StarOffice. Can you test with StarOffice and see if the correct behavior is being observed?
Remus: I used the Java Demo StylePad to reproduce the bug. StylePad is a Styled TextEditor. Simplest thing I did was to set the magnifier to 4X, ensure that we start at the top of the application window, and that the magnifier is not able to see the bottom of the textarea, and then start typing in StylePad, (pressing RETURN alot). Eventually, as you're typing the cursor moves past the end of the text visible at the bottom of the magnifier, I'm still typing but the magnifier isn't magnifying the area I'm typing. NOTE that this is prior to a change I made to produce text-changed:(insert|delete) events around Feb 6, so try use a build of the JABG with this disabled, if you need one let me know and I can provide it. Once I enabled text-changed events with the insert/delete qualifiers this went away, but the point of this bug was that I shouldn't have had to do this for gnopernicus to track the caret position using the magnifier.
Darren: does gnopernicus track the caret correctly when you only use the keyboard to move the caret, rather than when entering text?
Bill: Strictly speaking, I am only using the keyboard... But if you mean only using the arrow keys, then I don't know since there's another bug where gnopernicus is consuming the arrow keys in Java apps, so I can't use them...
Darren, would you please bump up the priority of the 'captures arrow keys' bug? That one is serious!
Bill: Gnopernicus consuming the arrows keys is a side effect of registering a callback for same keycombinations. This callback is called NOT ONLY for registered keys. (See bug 105963 and 106081)
Darren: If I understood correctly, if text-changed events are emited this bug is not anymore present. It is present if text-changed events is not emited. This behaviour is correct for gnopernicus. If a caret-moved event is processed by gnopernicus, but the number of character for object who emitted it differs from previous, then this event is omited. (gnopernicus assumes that a text-changed event will came too). A workaround possibility exists, but it will be toolkit specific and has a side effect: gnopernicus will not be able to report the text inserted/deleted. (this text is event specific(see bug 95921)). Also in case in which both events (text-changed and caret-moved) will be emited, is imposible for gnopernicus to skip the caret-moved event which a side effect of text-changed event. In my opinion emiting always text-changed event is the best solution. Is any reason why Java apps doesn't do that? Please tell what I have to do to get text-changed events on my computer..
Hi Paul: I still do not understand why you are using (exclusively) text-changed events for magnifier tracking. What, for instance, if the user is only "reading" the text, or is moving around the text buffer using the arrow keys or via control-key sequences? In such cases the text caret will move but of course there will be no change events - but the user will still want the magnifier to track the text caret. So I suggest that tracking the text caret makes more sense that using text-changed events for magnifier tracking (magnifier tracking is, after all, the primary reason for our implementing caret-moved events). As for the problem with the arrow keys for Java applications, are you saying that this is a side-effect of the java bridge bug 106081? Which keys are you 'intending' to catch (as opposed to those you are getting) ? (please reply to this last question in bug 106081 http://bugzilla.gnome.org/show_bug.cgi?id=106081)
Bill: gnopernicus needs to do an event filtering. In case it doesn't do that, all at-spi listened events will be reported to the user. In case user use keys to navigate, only caret-moved events are emitted, event which need to be reported to user (this event is reported to user for Java app too, but only if user made no changes in text. This proves that gnopernicus listen for this event, and, in some cases it works. In case in which user make changes in text (insert or delete (with BACKSPACE)) gnopernicus get 2 events (text-changed and caret-moved). Reporting both events is not a good ideea. User intension was to change the text, not to move caret. So, ignore caet-moved event is a good ideea, user wants to get only text-changed event. If (from different reason) one event is missing, gnopernicus is not able anymore to filter events. This is the reason why most applications must emit SAME event. If gnopernicus get text-changed event, user will get a feedback of his action.
Remus: Are you saying that gnopernicus does event filtering based on intercepted key events (i.e. Backspace) ? That would be a very bad idea I believe. I think you can do what you want by catching both events (text-changed and caret-moved) and doing some comparisons between the caret position and the offset(s) of the text change. If you get the caret-move first, you can pre-empt the caret move speech (by either calling shutup or another means) when you get the text-changed event. If you get the text-change event first, you can compare the caret offset (from the move event) with the text-changed offsets to determine whether this caret move is significant (i.e. not associated with the most recent text-change from the object) or redundant (resulting from a text-change event to the same object). In this way it would not even be necessary for the events always to come in the same order. I am sure that with a bit of thought you can improve on this suggestion while still being fully general (i.e. not caring whether the user is "editing" text or just "viewing" it). Realize for instance that text-change events can come from object that the user isn't editing! So you must be able to handle these cases without trying to guess what the user is 'doing'. -Bill
I do NOT intercept key events. In my previous comments my intention was to show that caret-moved event is emitted in case of deleting text only if BACKSPACE is pressed. (if user pressed DELETE this event will not be emitted, caret will have same position). Logic described by you previously is already implemented in gnopernicus. The text-changed event I still belive that is neccesary. If user press DELETE key, then no event from Java app is emitted, which means NO feedback for user. Also, in case user only view a text, using arrow key, he prefers to hear the character at caret (character after caret symbol on screen). If he type, he want to hear the new text, not the text after caret. Logic in gnopernicus is made to be insensible to order in which events are caming (text-changed then caret-moved or caret-moved then text-changed). I dont think that this bug depends on bug 105963, so this dependency should be removed.
Hi Remus: let me see if I understand your points: #1) Java applications aren't emitting text-changed:delete events when the user presses DELETE; I agree that would be a bug. #2) in other cases the user wants to hear something different when text-changed events (along with caret-moved) events are generated, as compared to the case when only the caret is moving. #3) you are always listening to caret-moved events, so you can speak something appropriate as the user moved through a text document. Please let me know if I have misunderstood one or more of these points. Other than case #1 above, can you describe specific cases where you think text-changed events are not happenning when they should? Also, I think Darren has made some changes to text-changed events from the java-access-bridge recently (they now include the 'detail' string), so do you still see these problems with recent versions of the bridge? thanks -Bill
Bill: all 3 points are correct. I want to notice here that all my remarks for this bug are based on Darren words when he reported this bug. He said that this bug appears BEFORE changes made in JAB for emitting "text-changed:inset/delete" event. Gnopernicus assume that a text-changed event is emitted when text is changed (this is the corect behaviour for any application). Darren: what I have to do to get text-changed events from Java apps. I tried to get java-access-bridge from cvs, but no result. Do I have to do somethings special.
Remus: Shouldn't need anything special from CVS, simply cvs co java-access-bridge works for me... Ensure CVSROOT is : :pserver:<user>@cvs.gnome.org:/cvs/gnome As for text-changed events - I was dispatching text-changed events before, just without the :insert (or alternatives) part, ie: object:text-changed I still am of the opinion that in that case gnopernicus should still have acted upon the text-caret-moved event...
I reinstalled some packages and now I get "text-changed" event. If I understood correctly text-changed details (insert and delete) are not iplemented in JAB yet, but them will be. Am I right?
text-changed events have included the 'detail' substring since about Feb 6, according to Darren.
My question was in fact if I need to make changes in gnopernicus to listen for "text-changed" (which is different that "text-changed:insert/delete") to work with Java apps, or this event detail will part of event name. Also I observed that AccessibleTextChangedEvent_getChangeString returns always NULL for Java apps.
As for getChangedString - that's a known issue that I'm looking into... No I don't think you need to listen on text-changed events, but what I don't understand is why the magnifier isn't following the cursor regardless of text-changed events, surely the caret-moved events should be the one that influences where the magifier is showing? In other words, where the caret is currently located should be main point of interest when within a TextArea. Naturally if the mouse moves, or focus changes then things change, but within the TextArea it's the text-caret-moved that's vital. Of course if a text-changed event occurs in a location other than where the caret is, then that will be of interest, but only in that case...
text-changed events now include the detail string as Darren says. Remus, I thought you told me earlier this week that gnopernicus _is_ listening to text-caret-moved events. Darren seems to think that either is not true, or it's not working properly. The reason for thinking that is this: gnopernicus is not responding when the user deletes text in such a way that the text-changed event is not properly sent, we agree that a missing or incorrect text-changed event would be a bug. BUT java applications *are* sending text-caret-moved events, so gnopernicus should be doing _something_ in these cases even if it's not strictly what the user expects. The fact that gnopernicus is not responding to text-caret-moved events in such cases indicates a gnopernicus bug of some kind. Does that make it clearer?
Currently, gnopernicus is listening some at-spi events. It processes them (filtering, transforming, rejecting) and present to user only a part of these events. In case in which "caret-moved" is not an event generated as a user intension only to move the caret it will be rejected. That's the reason why gnopernicus doesn't react to it. Gnopernicus listens this event. If test are made with gtk application a correct behaviour could be observed.
this still doesn't add up. The initial bug report indicated that gnopernicus was not speaking in a case where both text-changed:delete and text-caret-moved events were expected. It was determined that only text-caret-moved events were being sent correctly... but in that case gnopernicus should certainly have been responding to those events which were intercepted! I.e. if gnopernicus wasn't getting correct text-changed events, then how could it have been correct in rejecting the text-caret-moved events?
Darren: Remus reports that text-changed:insert events are not being fired by StylePad when the user enters text. Can you confirm/investigate?
Bill, I've just found that I missed the fix for the qualified text-changed events - I put back half the fix not it all, so that's why the event isn't received by Remus. I'm putting the rest back now. On that note, whether I'm generating these events should (as you've said in your previous comment) not be relevant, the problem is that somehow gnopernicus still isn't following the caret-moved events as you'd expect.
Peter reports that gnopernicus' magnifier now appears to be correctly following the text caret in StylePad, so the original test case may no longer be reproducible. Darren, are you aware of any remaining symptoms?
I can verify that I'm now seeing tracking of the caret position as I would have expected. Thanks Remus. I've not seen any other side effects, so I guess it's OK to mark this bug as resolved.
Created a solution using toolkit name. Solution commited on CVS.
Remus: What do you mean about a solution "using toolkit name" ? Darren reported that this bug was fixed and didn't need further work )see previous comment). Also, we really should NOT have any toolkit-name-specific code in gnopernicus, so this fix should NOT have gone into CVS! Please respond. thank you.
"Using toolkit name" means that when gnopernicus receives a "caret-moved" event it looks at the toolkit name for that object. If the name is "J2SE-access-bridge" the event is always reported (the event is never rejected). In order to have no toolkit-name-specific code in gnopernicus all toolkit should have same behaviour. That means to emmit same events for same user actions in same conditions. But, for now is no uniform behaviour for all toolkits.
Remus: I understand the intent of the code you added, but I think it should be removed. Our goal is to achieve the same behavior between toolkits wherever possible, and I do not think we have given up on this for the current case. PLEASE give us a specific reproducible test that explains what different behavior you are seeing between the two toolkits, this is still not clear. Once we have this test case we will see what we can do about making the behavior the same.
Remus: As far as I can see the code that's somehow not able to handle Java Apps is below? Is that correct? if (info2.text_selections_count == info.text_selections_count && info2.text_selections_count == 0 && info2.text_caret_offset != info.text_caret_offset && info2.text_char_count == info.text_char_count) {} else { process = FALSE; } If that's correct, I can't see how moving the cursor around using the arrow keys is failing on this. Previously this might have been a problem because the keys were being consumed by gnopernicus (see bug 106081), but that's not the case anymore (As far as I can see) so this test should passing. In fact I commented it out rebuilt the libsrlow and the magnifier was still following the caret position. As I said before, I think this but can be marked as resolved.
Darren: if the text is already typed or you move to other paragraph and you navigate using arrow keys everything is OK. But if you type a new text and then you navigates using arrow keys magnifiers is not working properly (in fact logic indicated by you will reject all caret-moved events), so this bug is not resolved. Bill: if you open a gtk application which has an entry and type a text and look at emmited events using at-spi event-listener-test test program and then repeat same scenarios for SO you will see the differences.
Darren, have you carried out the test Remus posted? What is the status of this bug?
I would like to conclude what is happening in this bug: 1. The initial scenario is NOT an issue anymore: the magnifier tracks caret position for StylePad application (and all other JavaToolkit applications). OVERVIEW: The bug was caused by the lack of text-changed:delete|insert. Now these events and their details are provided with JABG so, for Java applications too, and gnopernicus' logic deals with them like it does with other applications (it is the same logic that treats text-chaged and caret-move in GTK Toolkit applications and Java Toolkit applications - warkarounds were removed in today's commit of gnopernicus) TEST CASE (that I used): - gnome-2.2 from 10th of April, java-access-bridge from today's CVS, gnopernicus from today's CVS - start StylePad - start gnopernicus with the magnifier service on - start simple-at test program attached to this bug (this is optional) - type text - move with arrows in a line - delete text with Delete - delete text with Backspace ACTUAL RESULTS = EXPECTED RESULT Events are fired corectly and gnopernicus-magnifer is tracking corectly the caret: - type text: text-changed:insert and caret-moved - move with arrows: caret-moved - delete text with Delete: text-changed:delete - delete text with Backspace:text-changed:delete and caret-moved ADDITIONAL INFORMATION (a) Screenshot attached : simple-at test program inspecting events of SwingSet2 (other Java Toolkit application that I've tested) (b) simple-at test program (modified and attached) proves that events are comming as they should. 2. The bug remains valid for StarOffice, BUT it depends on bug #13677 from IssueZilla. In fact this is NOT a gnopernicus bug, but a StarOffice bug.Please see: http://www.openoffice.org/issues/show_bug.cgi?id=13677 3. Key problems (arrows or not) are not the object of this bug. 4. New bug needs to be opened for AccessibleTextChangedEvent_getChangeString which is returnig always NULL for Java Toolkit applications.
Created attachment 15931 [details] ScreenShot : simple-at inspecting events from SwingSet2
Created attachment 15932 [details] [review] test program (needs to be copied in at-spi/test to be compiled)